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ABSTRACT 


The systems engineering technical review (SETR) process should be an event- 
driven process that consists of one or more programmatically independent reviews with 
defined entrance and exit criteria. The SETR assesses the technical health, requirements 
accuracy, design maturity, testing effectiveness, and sustainment support over the 
program life cycle. This thesis focuses on how to tailor the SETR implementation to 
improve the program’s return on investment (ROI). To address the research question, a 
literature review was performed focusing on SETR-related policy, instructions, and 
memoranda and a survey was conducted of subject matter experts. Eeveraging a tailored 
SETR implementation provides the necessary structured engineering framework while 
keeping up with a dynamic software development environment to meet the increasing 
need for enhanced capability delivered to the warfighter in a shorter timeframe. More 
than 80% of the survey respondents indicated tailoring was occurring within programs to 
address specific needs such as leveraging an agile software development model. Eactors 
external to the organization continue to be the primary obstacle no matter the program 
size. Euture naval software acquisition programs should engage leadership, focus on 
preparation, maintain communication early and often, and educate stakeholders to 
improve the ROI of the tailored SETR implementation. 
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EXECUTIVE SUMMARY 


According to the Naval SYSCOM Systems Engineering Policy (2009), the 
systems engineering technical review (SETR) process should be an event-driven 
process that consists of one or more programmatically independent reviews with 
defined entrance and exit criteria. The SETR reviews assess the technical health, 
requirements accuracy, design maturity, testing effectiveness, and sustainment support 
over the program life cycle. The jointly signed Systems Command (SYSCOM) policy 
states that the program SETR events along with the event entrance and exit criteria are 
documented in the Systems Engineering Plan (SEP), which is signed by the program 
Milestone Decision Authority or other designated authority based on the program 
Acquisition Category level. The SYSCOM policy notes that event closure normally 
occurs only after the exit criteria has been met, but the SETR technical review board 
(TRB) chair must concur with the identified action items along with any plan of action 
and milestones and/or mitigations. According to the policy, the SETR output ultimately 
informs the program manager (PM) if the program is technically ready to move on to 
the next phase of the acquisition process. 

Prom a policy perspective, the Department of Defense (DOD) and Department 
of the Navy (DON) directives and instructions lay the systems engineering foundation 
for the naval SYSCOM guidance regarding SETR implementation. In addition, the 
Navy has emphasized the need to deliver capability versus systems, and acquisition is 
impacted by this capability vision requiring innovative application of SETR for the 
system of systems (SoS) or platform level efforts, which traditional SETR is not well 
structured to support. This thesis provides an explanation of the traditional SETR 
process implementation within the various naval SYSCOMs. Additionally, this thesis 
provides an overview of how the SYSCOMs enable major acquisition programs and 
other projects to tailor the SETR implementation to fit the cost, schedule, and 
performance addressing program variables such as new software development 
methodologies and the Navy’s capability delivery vision. This research includes 
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recommendations on how to increase likelihood of a program return on investment 
(ROI) for a tailored SETR implementation. 

Leveraging a tailored SETR implementation provides the necessary structured 
engineering framework while keeping up with a dynamic software development 
environment to meet increasing need for enhanced capability delivered to the 
warfighter in a shorter timeframe. However, how can the SETR implementation be 
tailored to most efficiently leverage resources and minimize schedule impact while 
addressing the acquisition, technical, and policy/legal requirements within naval 
software acquisitions? To accomplish this research, an empirical research methodology 
was leveraged consisting of four phases. Eirst, a literature review was performed 
primarily focused on SETR-related policy across the DOD, DON, and naval 
SYSCOMs. The next phase consisted of an electronic survey with program system 
matter experts (SME) who had key leadership roles within various sizes of programs. 
Next, data from the survey responses and key findings from the literature review were 
analyzed together to identify commonalities and differences as it relates to the research 
questions. Einally, research conclusions and recommendations for future tailored SETR 
implementations were addressed as well as any suggestions to extend this thesis 
research. 

More than 80% of the survey respondents from across various DON 
organizations indicated tailoring was occurring within programs to address program 
specific needs such as aggressive schedule and leveraging an agile software 
development model. The tailored SETR events are directly impacting the program’s 
next phase through adjustment of technical design and other key program variables. In a 
tailored SETR implementation, the programs find the most ROI by tailoring entrance 
criteria, exit criteria, and/or which specific reviews are included. This holds true for 
both commercial off-the-shelf and new development software acquisition models. 
Eactors external to the organization continue to be the primary obstacle, no matter the 
program size, for determining the ability to successfully tailor and implement the SETR 
process. Euture naval software acquisition programs should engage leadership, focus on 
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preparation, maintain early and often communication, and educate stakeholders to 
improve ROI of the tailored SETR implementation. 
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I. 


INTRODUCTION 


A. BACKGROUND 

The systems engineering technical review (SETR) process should be an event 
driven process that consists of one or more programmatically independent review events 
with defined entrance and exit criteria (Department of Navy Space and Naval Warfare 
Systems Command [DON SPAWAR] 2009, 7). The SETR reviews assess the technical 
health, requirements accuracy, design maturity, testing effectiveness, and sustainment 
support of the program being reviewed over its life cycle. The jointly signed Systems 
Command (SYSCOM) policy states that program SETR events along with the event 
entrance and exit criteria are documented in the Systems Engineering Plan (SEP), which 
the SEP is signed by the program Milestone Decision Authority (MDA) or other designated 
authority based on the program Acquisition Category (ACAT) level. The SYSCOM policy 
notes that event closure normally occurs only after the exit criteria has been met, and the 
SETR technical review board (TRB) chair concurs with the identified action items along 
with any plan of action and milestones (POAM) and/or mitigations. According to the 
policy, the output of a SETR event ultimately informs the program manager (PM) whether 
the program is technically ready to move on to the next phase of the acquisition process. 

The SETR process is intended to align with the phases of the acquisition process 
although the alignment varies depending on the acquisition model being executed. The 
models are detailed in the Department of Defense (DOD) Operations of the Defense 
Acquisition System (2015). Erom a software acquisition perspective, the DOD recognizes 
four models to which major ACAT programs align, which are software intensive, 
incrementally deploying software, accelerated acquisitions, or a hybrid acquisition with 
software being the dominate component (Department of Defense [DOD] 2015). 

A traditional SETR process can include upwards of 12 types of reviews such as a 
system requirements review (SRR), preliminary design review (PDR), critical design 
review (CDR), and test readiness review (TRR). Various individuals are involved in these 
reviews to include program engineers, technical subject matter experts (SME), and other 
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independent stakeholders to ensure their areas are appropriately addressed within the 
program (DON SPAWAR 2009). For example, the stakeholders in a TRR event would 
include individuals from the developmental and/or operational testing agency. Figure 1 
shows the timing of traditional SETR events in relation to the program acquisition phases. 
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Figure 1. Traditional SETR Model. Source: DON SPAWAR 

(2009, Enclosure [1]). 


B. PURPOSE 

Leveraging a tailored SETR implementation provides the necessary structured 
engineering reviews while keeping pace with a dynamic software development 
environment to meet the increasing need for enhanced capability delivered to the 
warfighter in a shorter timeframe. In addition, the Navy has emphasized the need to 
deliver capability versus systems, and acquisition is impacted by this capability vision 
requiring innovative application of SETR for the system of systems (SoS) or platform 
level efforts, which tradition SETR is not well structured to support. This thesis provides 
an explanation of the traditional SETR process implementation within the various DON 
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SYSCOMs after the release of the memorandum from the Office of the Assistant 
Secretary of the Navy (Research, Development & Acquisition) (ASN[RDA]) in June 
2008 directing this process implementation. Additionally, this thesis will provide an 
overview of how the SYSCOMs enable major acquisition programs and other projects to 
tailor their processes to fit the cost, schedule, and performance to address program 
impacts such as new software development methodologies and the Navy’s capability 
delivery vision. This research includes recommendations on how to increase likelihood of 
a program’s positive return on investment (ROI) for a tailored SETR implementation. 

C. RESEARCH QUESTIONS 

The primary research question for this thesis is: How can the SETR 
implementation be tailored to most efficiently leverage resources and minimize schedule 
impact while addressing the acquisition, technical, and statutory and regulatory 
requirements within naval software acquisitions? To answer this question, the following 
subsidiary questions will also be addressed. 

1. What are the obstacles (e.g., policy, timelines, and maturity) and risks faced 
by naval programs that have attempted to tailor the SETR process? Do these 
obstacles vary based on the size of the program (e.g., ACATI vs non 
program of record)? 

2. What are the critical elements of the current system engineering technical 
review process (e.g., entrance/exit criteria, order of reviews, risk 
assessment) that program managers and/or lead engineers should include in 
a tailored implementation? What incentives or return on investment do these 
critical elements provide? 

3. Are there any considerations/differences that should be accounted for when 
tailoring the process to support various acquisition models[(e.g., 
commercial off-the-shelf (COTS), new development, government off-the- 
shelf (GOTS)]? 

4. What can engineers and program managers of future naval acquisitions 
learn from other programs that have attempted to tailor? 

D. BENEFIT OF STUDY 

This research includes recommendations based on lessons learned on how to 
increase likelihood of a program ROI for a tailored SETR implementation as it relates to 
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naval software acquisitions. The recommendations will assist program leadership in 
making better decisions on where to allocate software engineering resources within the 
schedule and funding constraints. While the thesis is focused on the DON, the 
recommendations are applicable to SETR implementations for software acquisitions 
programs across the broader DOD. 

E. METHODOLOGY 

To accomplish this research, an empirical research methodology was used in four 
phases. First, a literature review was performed focused on SETR-related policy, 
instructions, and memoranda across the DOD, DON, and naval SYSCOMs. The literature 
review also included other documentation (e.g., standards) from industry organizations 
such as Institute of Electrical and Electronics Engineers (IEEE). The next phase consisted 
of an electronic survey with program SME who had key leadership roles (e.g., program 
manager, lead engineer) within various sizes of programs (e.g., ACAT I, project). Table 4 
includes the survey questions along with the potential responses. Next, data from the 
survey responses and key findings from the literature review were analyzed together to 
identify commonalities and differences as it relates to the research questions. Finally, 
research conclusions and recommendations specific to programs who wish to tailor SETR 
implementation were addressed as well as some suggestions to extend this thesis research. 

F. THESIS ORGANIZATION 

The chapters of the thesis are organized as follows: Chapter 1 is the introductory 
and background information. Chapter II is an overview of the SETR implementation which 
includes potential ways in which the process could be tailored, SETR success indicators, 
and applying SETR to support program risk mitigation. Chapter III is a review of the DOD, 
DON, naval SYSCOM policies, and industry standards relating to the SETR process. 
Chapter IV presents the data from the survey. Chapter V is the research analysis and 
discussion tying the policy review presented in Chapter III to the survey data with the goal 
of addressing the research questions. Chapter VI is the conclusions from the research and 
suggestions for follow on research. 
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II. SYSTEMS ENGINEERING TECHNICAL REVIEW 

IMPLEMENTATION 

A. INTRODUCTION 

ASN (RDA) directed in a 13 June 2008 memorandum that the SYSCOMs develop 
and implement the SETR process within their commands no later than 120 days from the 
date of memorandum to “ensure appropriate systems engineering aspects are included in 
the Gate Reviews” (1). The SETR process, which aligns to the acquisition phases, has 
since been documented in various SYSCOMs instructions and other supporting 
documentation as the implementation has matured. According to the jointly signed 
SYSCOM policy titled Naval SYSCOM Systems Engineering Policy, SETRs are “event- 
driven, have specific entry and closure criteria, and are used to evaluate technical 
baselines. The SETRs are chaired by technical authorities independent of the program, 
with participation by the PM and support from the associated Integrated Product Team 
(IPT) (govemment/contractor)” (DON SPAWAR 2009, 7). The SYSCOM policy states 
the SETR process assesses the program’s technical health, requirements accuracy, design 
maturity, testing effectiveness, and sustainment support over the program life cycle. The 
SETR events that the program executes along with the event entrance and exit criteria are 
documented in the SEP, which is signed by the program MDA or other designated 
authority based on the program’s AC AT level. Closure of the event normally occurs only 
after the exit criteria has been met; however, the SETR TRB chair can close an event 
with outstanding action items as long as the chair concurs with the identified action items 
along with any POAM and/or mitigations. According to the joint SYCOM policy, the 
output of a SETR event ultimately informs the PM if the program is technically ready to 
move on to the next phase of the acquisition process. 

The Naval Sea Systems Command (NAVSEA) Research & Systems Engineering 
Warfare Systems Engineering and Human Systems Integration Directorate (OSH) 
includes this comprehensive list of SETR objectives in the December 2009 Technical 
Review Manual: 
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Assessing the development maturity based on technology maturity and 
technical development goals from the SEP, SE events and 
accomplishments, and empirical test data supporting progress to date. 

Ensuring operational, functional, performance. Information Assurance 
(lA) and cost requirements, designs, technical performance measurements, 
and technical plans are being tracked and are on schedule. 

Assessing the system requirements to ensure that the requirements are 
unambiguous, consistent, complete, feasible, verifiable, ranked for priority 
and stability, and traceable to top-level requirements (TERs). 

Demonstrating that the relationships, interactions, interdependencies, and 
interfaces between required items and externally interfacing items, system 
functions, subsystems, and system elements (including usability and 
maintainability), as appropriate, have been addressed. 

Ensuring open architecture (OA) in the emerging system; assessing the 
degree of Naval Enterprise reuse (availability and potential for component 
reuse); and critiquing any tradeoff decisions. 

Ensuring that the results of trade studies are used to define concepts and 
that risks associated with alternatives have been analyzed. 

Ensuring that technical designs will be usable by the target warfighter 
population, meet the Elect requirements and have viable training options. 

Ensuring that trade studies include lA and safety requirements and that the 
risks of failing to meet these requirements have been properly treated. 

Confirming that the effects of technical risks on cost, schedule, and 
performance, as well as risk reduction measures, rationale, and 
assumptions made in quantifying the risks have been addressed. 

Providing a forum for communication, coordination, and integration 
across all acquisition disciplines. 

Establishing a common configuration baseline from which to proceed to 
the next level of design. 

Confirming that continued development is warranted (with or without 
modifications to the program), and when it is not, identifying the technical 
measures necessary to preserve for reuse as much of the technology, 
hardware (HW), and software (SW) developed to date as possible. In the 
case where program redirection or restructuring is considered appropriate, 
the review should ensure that executable alternatives have been defined 
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(discontinue development, or take corrective action on the products and/or 
processes of the current phase before proceeding to the next phase). 

Verifying the system is ready for the appropriate level and type of testing 
or that appropriate approvals for production and certification have been 
granted 

Identifying resources (e.g., people, funding, support assets, test facilities, 
as appropriate) required for continued development, testing, or production. 

Reaching technical concurrence with stakeholders. 

Provide a check of proposed design configuration versus specification and 
contractual requirements. 

Evaluate systems configuration at different stages in terms of 
requirements. (Department of Navy Naval Sea Systems Command [DON 
NAVSEA] 2009, 3-2-3-3) 

To accomplish these objectives, various individuals are involved in the reviews to 
ensure their areas of expertise are appropriately addressed within the program. Key 
participant roles that are consistent across the SYSCOM implementations include: SETR 
chair(s), independent assessors, and integrated product team (IPT) participants. The 
responsibilities vary at the SYSCOM level for each of these key participant roles, but 
generally only to account for their organizational construct and do not represent a 
substantive variation. 

The SETR TRB chair position can be held by one or more persons if there is 
significant interest in the program itself and/or perhaps a significant partner that has 
interest in the outcome of the system being delivered. Based on review of all SYSCOMs 
SETR policies, the minimum responsibilities of the chair(s) include: 

• concurring on the entrance and exit criteria for the review 

• facilitating collaborative participation throughout SETR process 

• providing viable technical recommendation/guidance to address risks and 
issues identified in the process 

• elevating any technical issues that cannot be resolved by SETR 
participants to SYSCOM Chief Engineer (CHENG) 

• signing the SETR report (DON SPA WAR 2009, 9-12) 
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The SETR event independent assessors are either a Technical Authority (TA) 
related to the area being covered or, at a minimum, a designated representative of the 
SYSCOM TA. In some cases, the TA might also be asked to co-chair the SETR event. 
Erom a responsibility perspective, the independent assessors will focus their efforts on 
assessing risk in their area of expertise as well as providing viable technical 
recommendations/guidance to address these risks. 

Beyond the SETR TRB chair and independent assessors, the IPT participants are 
key members of the SETR review team and make up the bulk of the membership 
especially from an execution of work perspective. These individuals review the artifacts 
associated with the SETR event to validate that the documents align to the higher level 
naval guidance requirements, do not conflict with one another, and address the intent of 
the review and the overall program (DON SPAWAR 2016c). The risk and issues are 
identified on the basis of the work from these members, which will ultimately turn into 
action items if not addressed prior to the closure of the SETR event as discussed in the 
SYSCOM policy. Eigure 2 provides a SYSCOM level example of how the roles 
organizationally interact. Participants include a TRB, SETR support team (SST), and 
independent assessment team (lAT). The participants ultimately report back to the PM 
and CHENG. 
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Figure 2. Example SYSCOM SETR Organization Chart. 

Source: DON SPA WAR (2016c, 8). 

The primary traditional SETR implementation events include the following: 

• Critical Design Review (CDR) 

• Eunctional Configuration Audit 

• Integration Readiness Review 

• Operational Test Readiness Review 

• Physical Configuration Audit 

• Preliminary Design Review (PDR) 

• Production Readiness Review 

• Software Specification Review 

• System Eunctional Review 

• System Requirements Review 

• System Verification Review 
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Test Readiness Review (Department of Navy Marine Corps Systems 
Command [DON MCSC] 2014). 




Each event aligns to a different acquisition life cycle phase and determines the programs 
technical readiness to proceed into the next phase. Generally, entrance and exit criteria 
are fulfillment of a checklist of documentation and key program elements (e.g., 
verification of completeness of test procedures). The entrance criteria ensure the program 
is sufficiently technically mature to enter the event. SETR events are resource intensive, 
so premature entry will only frustrate all participants (at best). Similarly, the exit criteria 
ensure the program is technically ready to move to the next program phase. Examples of 
both entrance and exit criteria are shown in Table 1. 


Table 1. Example SETR Entrance and Exit Criteria. 

Source: DON SPAWAR (2016c, 48). 


Theme 

Event 

Proposed Update (June 2016) 
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B. PROCESS TAILORING 


Each SYSCOM has varying guidance regarding tailoring. Some SYSCOMs only 
lightly touch on the subject requiring that the tailoring is documented and agreed to 
within the program’s SEP, while others provide more specific models and guidance. Eor 
example, SPAWAR’s Systems Engineering Technical Review Organizational Standard 
Process Handbook provides specific tailored SETR models such as the Accelerated 
Acquisition Model. Similarly, but in the commercial sector, the IEEE Standard 15288, 
which is titled Systems and software engineering—Systems life cycle processes, has an 
appendix focused on process tailoring that is extremely applicable to many DOD 
acquisition programs. IEEE Standard 15288.2, which is titled Technical Reviews and 
Audits on Defense Programs, gets into SETR events and amplifies IEEE 15288. When 
determining if tailoring the SETR implementation is beneficial, it is important to be able 
to articulate the purpose, outcomes, and activities/tasks associated with the tailoring, 
which are the key points of IEEE Standard 15288 Annex A. With regards to the 
activities, the program should be able address the following items detailed in IEEE 
Standard 15288: 

Identify and record the circumstances that influence tailoring. These 
influences include, but are not limited to: 

Stability of, and variety in, operational environments 

Risks, commercial or performance, to the concern of interested 

parties; 

Novelty, size and complexity; 

Starting date and duration of utilization; 

Integrity issues such as safety, security, privacy, usability, 

availability; 

Emerging technology opportunities; 

Profile of budget and organizational resources available; 

Availability of the services of enabling systems; 


11 



Roles, responsibilities, accountabilities and authorities in the 

overall life cycle of the system; 

The need to conform to other standards. 

In the case of properties critical to the system, take due account of the life 
cycle structures recommended or mandated by standards relevant to the 
dimensions of the criticality. 

Obtain input from parties affected by the tailoring decisions. This 
includes, but may not be limited to: 

The system stakeholders; 

The interested parties to an agreement made by the organization; 

The contributing organizational functions. 

Make tailoring decisions in accordance with the Decision Management 
process to achieve the purpose and outcomes of the selected life cycle 
model. 

Select the life cycle process that require tailoring and delete selected 
outcomes, activities, or tasks. (IEEE Computer Society 2015, 86-87) 

These activities are consistent with the principles within the SYSCOMs’ guidance 

as well. Eor instance, SPAWAR’s Systems Engineering Technical Review Organizational 

Standard Process Handbook defines tailoring as that which is “based on sound systems 

engineering practices, program or project complexity, and risk level, while meeting the 

objectives of the identified event” (2016c, 4). This definition ties closely to the IEEE 

Standard 15288 “identification and recording of the circumstances that influence the 

tailoring” as well as defining the purpose and outcome of tailoring the process itself 

(IEEE Computer Society 2015, 86). NAVSEA’s 05H Technical Review Manual directs 

that tailoring be “consistent with good engineering judgment and program complexity 

and risk levels” (2009, 3-6). Naval Air Systems Command (NAVAIR) Instruction 

4355.19E, which focuses on the Systems Engineering Technical Review Process, 

reinforces similar concepts to those of both SPAWAR and NAVSEA do; however, 

NAVAIR goes a step further to explain that “tailoring takes the form of deletion (removal 

of reviews and elements not applicable), alteration (modifying and combining reviews 

and elements to more explicitly reflect the application to a particular effort) or addition 
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(adding reviews and elements to satisfy program elements)” (2015, 6-7). Even with the 
additional context, this NAVAIR Instruction 4355.19E is still very consistent with the 
IEEE Standard 15288 process tailoring. 

C. METRICS 

The International Council on Systems Engineering (INCOSE) published a 
Systems Engineering Leading Indicators Guide on 29 January 2010, which defines a 
leading indicator as “a measure for evaluating the effectiveness of how a specific activity 
is applied on a project in a manner that provides information about impacts that are likely 
to affect the system performance objectives” (6). This is a critical definition at the heart 
of SETR events as the technical leadership of the program should be looking at any 
leading indicators (specific sets of metrics) across the program that would indicate 
impacts that would likely affect the systems performance or ability to meet the 
performance objectives laid out in the system requirements or other core documentation 
(e.g.. Capability Development Document). Appendix A gives additional details regarding 
sample leading indictors based on IEEE Standard 15288. 

Eeading indicators generally are more focused on predictive analysis to support 
technical leadership decisions. However, it is important to not ignore conventional 
measures within the SETR event to ensure the technical report, which is an output of the 
SETR, provides a holistic look at the health and includes best recommendations to the 
PM. Table 2 includes some sample metrics that are included in SPAWAR’s Systems 
Engineering Technical Review Organizational Standard Process Handbook. Several of 
the metrics are trending related around tracking risk and issues (R&I) (e.g., number of 
R&I by program) that would be measured over multiple SETR events. Using metrics 
such as those in the handbook provide a continuous process improvement aspect to the 
SETR event in order to minimize the potential of it being executed as a simple check in 
the box. In addition, the metrics enable a mechanism to assess maximum value from the 
program office perspective and/or a retrospective look back to improve in the next review 
event if the event did not provide value relative to the resources expended. 
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Table 2. Sample Metrics. Source: DON SPAWAR (2016c, 7). 


Quantitative 

Trends 

Demand Signal 

Nuinber of SETR by 

Type, Maturity, and 

PKW 

Number of R&I by 

PEO, PMW, Program 

Workload forecast 

Number of R&I by 

Closed, Open with 
Mitigation, Accepted 
Risk/Issue with No 
Further Action 

By Assessment Area 

Loading 

Number of artifacts 
provided initially 

Number of R&I by 

SETR Type 

Planned events vs 

Actual events as 
documented in 

IMS, SEPs 

Number of artifacts 
provided after initial 
upload to TRP 

Number of R&I 

elevated to RFA 

Specialized 
reviewers needed 

Number of artifacts 
requested additionally 
during review period 

Number of R&I closed 
during review 



Additionally, ASN(RDA)’s Guidebook for Acquisition of Naval Software 
Intensive Systems published in September 2008 highlights the SETR process as a risk 
management tool as it can be used to verify the following: 

Requirements for the acquired system are defined (including granularity to 
address software-specific requirements); 

Requirements are transformed into an effective system (including effective 
software components and subsystems); 

The processes are consistent and repeatable for the entire life cycle, where 
necessary; 

The provider/supplier uses a systematic approach in providing the required 
products and services; 

The test resources (personnel, facilities, test assets, test plans, ranges, etc.) 
are available and the product is ready for test; 

The product is ready for Operational Evaluation (OPEVAE) (validation 
environment. Operational Test & Evaluation Eorce (OPTEVEOR) 
personnel, ranges, etc.); 
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Services/products are sustainable throughout the life cycle (including an 
adequate software support activity); and 

Systems are properly disposed of when they are retired from service. 

(Office of the Assistant Secretary of the Navy [Research, Development & 
Acquisition] 2008a, 6-10) 

Each of these areas highlighted in the ASN(RDA) guide are key areas that the IPX 
participants or SMEs would be evaluating as they review various documents that are part of 
the entrance criteria, exit criteria, and/or other documents that are requested as part of 
requests for information (REI) or requests for action (REA) during the event itself. By tying 
the leading indicators, metrics and risk mitigations together a program can ensure the 
reviews provide measurable ROI to the program office as it moves forward in delivering 
new capability to the warfighter. 

D. SUMMARY 

ASN (RDA) Memorandum from 13 June 2008, which is titled Systems 
Engineering Technical Review Process for Naval Acquisition Programs, directed the 
SYSCOMs develop and implement the SETR process within their commands no later 
than 120 days from the date of memorandum. The SETR process, which aligns to the 
acquisition phases, has since been documented in various SYSCOMs instructions and 
other supporting documentation as the implementation has matured. The SETR process 
should be an event-driven process that consists of one or more independent review events 
to assess the program’s technical health, requirements accuracy, design maturity, testing 
effectiveness, and sustainment support over the program life cycle (DON SPAWAR 
2009, 7). The selection of SETR events to include any intended tailoring that the program 
executes is documented in the SEP. Per the jointly signed SYSCOM policy, the output of 
a SETR event ultimately informs the PM if the program is technically ready to move on 
to the next phase of the acquisition process. The following chapter provides a review of 
the DOD, DON, naval SYSCOM policies, and industry standards that give direction and 
guidance related to the process, entrance/exit criteria, leadership involvement, tailoring, 
and all other applicable aspects of the SETR process. 
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III. POLICY AND STANDARD REVIEW 


A. INTRODUCTION 

This section provides a review of the DOD, DON, naval SYSCOM policies, and 
industry standards that give direction and guidance related to the process, entrance 
criteria, exit criteria, leadership involvement, tailoring, and all other applicable aspects of 
the SETR process. Each policy or standard will be summarized with an overview of how 
it relates to the SETR process, highlight any strengths of the document, and what if any 
possible unanticipated outcomes could be triggered by the policy or standard itself. Table 
3 includes the list of items that will be reviewed as part of this section. 


Table 3. Policy, Instructions, Memos Review Eist 


DOD Policy, Instructions, & Memos 

The Defense Acquisition System (DODD 5000.01) 

Operations of the Defense Acquisition System (DODI 5000.02) 

Business System Requirements and Acquisition (DODI 5000.75) 

Deputy Assistant Secretary of Defense for Systems Engineering (DODI 5134.16) 

DON Policy, Instructions, & Memos 

ASN RD&A Memo, SETR Process for Naval Acquisition Programs, 13 June 08 

Naval SYSCOM Systems Engineering Policy (SPAWARINST 5000.1) 

DON Implementation and Operation of the Defense Acquisition System and the Joint Capabilities 
Integration and Development System (SECNAVINST 5000.2E) 

SYSCOMs Policy, Instructions, & Memos 

NAVSEA 05H Technical Review Manual, Version 2.0, 18 December 2009 

NAVAIR Systems Engineering Technical Review Process (NAVAIRINST 4355.19E) 

NAVAIR Adapting Acquisition to Agile Software Development: A How-To Guide, Version 2.0, 19 
March 2014 

MCSC SETR Handbook (SIAT-HDBK-001) 

SPAWAR Systems Engineering Policy (SPAWARINST 5401.4) 

SPAWAR Systems Engineering Technical Review Policy (SPAWARINST 5400.3A) 

SPAWAR Systems Engineering Technical Review Organizational Standard Process Handbook 

Industry Standards, Instructions, & Memos 

IEEE Technical Reviews and Audits for Defense 

SEI Agile Software Teams: How They Engage with Systems Engineering on DOD Acquisition 
Programs 
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B. DEPARTMENT OF DEFENSE 


DOD Directive 5000.01, which is The Defense Acquisition System directive, 
establishes the basic policy governing how the DOD operates from an acquisition 
perspective to include responsibilities for key officials (e.g., the Assistant Secretary of 
Defense, Director of Operational Test and Evaluation), defining key terminology (e.g., 
program manager, milestone decision authority), and outlining policy around the Defense 
Acquisition System (e.g., flexibility, innovation) (DOD 2007). A key SETR-related 
element in this instruction is directing that the systems engineering approach should 
optimize “total system performance and minimize total ownership costs” (9). 
Eundamentally, the SETR process supports the key tenants of this systems engineering 
policy, so it is incumbent upon all involved in planning and executing the program SETR 
to ensure the reviews do not become a “check-the-box” event. 

DOD Instruction 5000.02, which is the Operation of the Defense Acquisition 
System instruction, establishes the policy for how acquisition programs shall be managed. 
This is the first DOD policy from a policy hierarchy perspective where SETR reviews 
begin to appear. Based on the AC AT of the program, enclosure 3 of this instruction lays 
out the minimum technical reviews that program managers will conduct, which consist of 
a PDR and a CDR (DOD 2015). The instruction enclosure also includes at what level 
these reviews will be conducted. Eor example per the instruction, MDAPS can conduct 
system level PDR assessments at the MDA level while an ACAT IC must conduct the 
review at the Component Acquisition Executive (CAE) level. 

DOD Instruction 5000.75, which is the Business Systems Requirements and 
Acquisition instruction, became effective on 2 Eebruary 2017 during this research effort. 
This instruction establishes the policy for how business systems will be managed (DOD 
2017). This instruction supersedes DODI 5000.02 for “business system acquisition 
programs that are not designated as a Major Defense Acquisition Program according to 
DoDI 5000.02” (1). A business system as described by this instruction include “financial 
systems, financial data feeder systems, contracting systems, logistics systems, planning 
and budgeting systems, installations management systems, human resources management 

systems, and training and readiness systems” (31). The instruction does not include 
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SETR-specific language, but does require technical and management assessments to be 
addressed in the required implementation plan. The “assessment structure will reflect the 
program’s tailored software development life cycle and may include events such as 
design reviews and test readiness reviews or short iteration retrospectives and acceptance 
reviews” (24). The SETR related policy and other documentation at the DON and naval 
SYSCOM level was written prior to this instruction, so expectation is that future 
iterations of these documents would address items such as SETR best practices and 
implications related to this instruction. 

DOD Instruction 5134.16, which is the Deputy Assistant Secretary of Defense for 
Systems Engineering (DASD(SE)) instruction, primarily establishes the roles and 
responsibilities of the DASD(SE) position. However, this instruction from a SETR 
perspective lays the foundation for the SEP including who needs a SEP and minimum 
items that should be covered (e.g., reliability growth, considerations for life cycle 
management and sustainability). This document is not overly prescriptive by outlining the 
minimum SETR items required in a SEP, which are expected across DOD from a 
consistency standpoint. 

C. DEPARTMENT OF NAVY 

ASN (RDA) Memorandum from 13 June 2008, which is titled Systems 
Engineering Technical Review Process for Naval Acquisition Programs, directed the 
naval SYSCOMs to develop and implement the SETR process within their commands no 
later than 120 days from the date of memorandum to “ensure appropriate systems 
engineering aspects are included in the Gate Reviews” (1). The 2008 ASN (RDA) 
memorandum also lays out other fundamental pieces of the process that will not change 
such as having including independent assessors, extended IPT members, and enabling 
tailoring based on program scope and complexity. Erom a naval prospective, this is a 
crucial piece of SETR documentation that lays the foundation, from which all the 
SYSCOMs’ SETR frameworks are derived. 

SPAWARINST 5000.1, which is the Naval SYSCOM Systems Engineering Policy, 
SYSCOM Engineering and Technical Authority Policy, is a jointly signed naval SYSCOM 
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policy that establishes systems engineering policy and a common SETR process across the 
DON (DON SPAWAR 2009). From a background perspective, the policy baselines roles, 
responsibilities, and definitions previously covered in other documents such as what the 
PM’s role is, what systems engineering means, and what the acquisition life cycle is. The 
policy states the SEP must detail the SETR schedule, which will help in understanding the 
sequencing relative to other program events. Once the SEP is approved, this will also 
indicate approval of the SETR schedule. In the section specific to SETR policy, 
SPAWARINST 5000.1 highlights the following fundamental SETR elements: 

• TRB Chair: must be designated in writing; generally a senior individual in 
the SYSCOM technical authority; holds the final authority for closing the 
review 

• Agenda: includes TRB membership, SETR participants, schedule, and 
entrance/exit criteria 

• Conduct: New issues should not arise at the SETR or this perhaps 
indicates that the SETR is being held prematurely 

• Document Review: depends on objective analyses, correctness and 
completeness, which should be measured against clearly stated objectives 

• Results: signed report by TRB chair closes out the event— must capture 
action items with appropriate status and include confidence level of 
baseline to move forward to next stage of development (10-12) 

SPAWARINST 5000.1 stays at a high enough level of policy to not hinder the individual 
SYSCOMs as they define their SETR frameworks and guidance to the acquisition and 
technical workforce. This document also provides baseline consistency and a common 
vocabulary for how SETRs are implemented across the DON, which is critical as the 
Navy drives towards more SoS deliveries requiring more cross SYSCOM interaction and 
program to program interdependencies. 

SECNAVINST 5000.2E, which is titled Department of the Navy Implementation 
and Operation of the Defense Acquisition System and the Joint Capabilities Integration and 
Development System, establishes the procedures for defense acquisition programs 
specifically related to the Joint Capabilities Integration and Development System (JCIDS) 
(Office of the Secretary of the Navy 2011). From a SETR perspective, the instruction 
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provides the basis for the concepts mentioned in the jointly signed SYSCOM policy 
SPAWARINST 5000.1, but it does reach a much broader audience being a SECNAVINST. 
Additionally, the instruction introduces a new “IT Box” concept that increases the value of 
the ability to tailor a SETR implementation. The “IT Box” approach, which is applied to 
software development programs, is “meant to lighten the burden of JCIDS as the program 
integrates systems enhancements described by the CDD [Capabilities Development 
Document], and allows programs to take full advantage of evolving commercial 
technologies” (1-11). In lieu of writing a Capabilities Production Document, the “IT Box” 
allows programs to write an annex to the CDD or an existing document to address four 
critical elements: “definition of threshold capability levels based upon today’s technology, 
a defined process for oversight and approval of future evolution, a defined plan for 
delivering those capabilities, and a defined level of funding” (1-11). The tailoring aspect of 
SETR is critical for those programs, which are leveraging “IT Box,” in order to not slow 
down the increased design to deployment schedule enabled by “IT Box.” 

D. SYSTEMS COMMANDS 

NAVSEA OSH Technical Review Manual published in December 2009 describes 
the SETR implementation objectives and execution guidelines (e.g., scheduling, 
documentation). The appendices of the document go into detailed descriptions of 
entrance and exit criteria for each SETR event, different documentation, architectural 
views, and scheduling details. Overall, the document is the most comprehensive of the 
SYSCOMs’ SETR-related documents with regards to execution of a traditional SETR. 
Erom a roles and responsibilities perspective, the NAVSEA SETR TRB chair, per the 
manual, consists of a minimum two co-chairs, which is a bit different than the other 
SYSCOMs. The co-chair consists of the PM and the independent TA, which can be 
joined by other co-chairs, if there is a significant partner that warrants the position as 
detailed in the manual. The responsibilities across the chairs are still consistent with what 
was previously described, just shared across all the chairs. The remainder of the SETR 
organization is also consistent with the previously described organization in the process 
overview. The NAVSEA OSH Technical Review Manual goes into great detail about 

initial planning for closing out the SETR event, which is depicted at a high level in 
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Figure 3 from the manual. These steps are repeated for each individual review (e.g., PDR, 
CDR) within the SETR implementation. The manual also includes opportunities and/or 
reminders throughout that the implementation should be tailored based on the scope and 
complexity of the program, but it does not give any models on how to do this as 
SPAWAR does, for example. This provides maximum level flexibility for program 
engineers with previous experience tailoring and/or motivation to seek out the 
opportunity. However, by not providing any best practices, models, and/or lessons 
learned, it allows programs to each independently make the same mistake, which could 
be costly in time, material, and/ funding to the DON. 
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Figure 3. Sample SETR Execution. 

Source: DON NAVSEA (2009, 3-15). 


NAVAIRINST 4355.19E, which is titled Systems Engineering Technical Review 
Process, establishes the policy, guidance, and responsibilities with respect to the SETR 
implementation for the command (Department of Navy Naval Air Systems Command 
[DON NAVAIR] 2015). The instruction, which was published in February 2015, starts 
very early introducing a new SETR event called the Release Backlog Review (RBR), 
which NAVAIR introduced to deal with the agile software development methodology. 
Eater in the instruction enclosure, NAVAIR includes entrance and exit criteria 
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expectations for incorporating this event into the SETR implementation. In addition, the 
NAVAIR instruction includes an entire section on SETR tailoring explaining that 
“tailoring takes the form of deletion (removal of reviews and elements not applicable), 
alteration (modifying and combining reviews and elements to more explicitly reflect the 
application to a particular effort) or addition (adding reviews and elements to satisfy 
program elements)” (6-7). Beyond this the NAVAIR instruction follows the general 
principles described in the SETR process overview, but the instruction takes a strong 
stance on encouraging tailoring when there is an “opportunity to optimize the program 
execution in the context of cost, schedule, and performance” (6). 

NAVAIR’s Adapting Acquisition to Agile Software Development: A How-to 
Guide, which was published in March 2014, documents some of the early concepts that 
were formali z ed in the NAVAIRINST 4355.19E. The guide gives more details in how to 
implement the RBR with some of the more typical SETR events such as PDR and CDR. 
RBR generally keeps a more consistent pace with what agile development refers to as 
development sprints, which require government SMEs to stay closely engaged. At some 
point after generally multiple RBR have been executed, a PDR will be held that will 
consist of the software architecture from these executed RBR and the “tentative allocated 
backlog will be established” (DON NAVAIR 2014, 7). RBR will continue to be held 
until the appropriate time for the CDR and likewise the TRR, which is shown in Eigure 4. 
Generally, NAVAIR programs are either Major Defense Acquisition Programs or Major 
Automated Information Systems including both HW and SW, which require the PDR and 
CDR. Otherwise, the program could tailor the SETR implementation further. 



Eigure 4. RBR within SETR Implementation. 

Source: DON NAVAIR (2014, 7). 
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Marine Corps Systems Command (MCSC) SIAT-HDBK-001, which is titled 
Systems Engineering Technical Review (SETR) Handbook, provides guidance on planning 
and execution of SETR events. The handbook sets the tone in the early stages of the 
document with a concise graphic that shows tailoring opportunities as it aligns to the 
entrance into the acquisition life cycle and program risk, which is included in Figure 5 
(DON MCSC 2014). It introduces a new event that the other SYSCOMs do not have, 
which is the non-developmental item (NDI) integration review (NIR). At a high level, the 
intent of the NIR, per the handbook, is to ensure that the system under review can meet the 
stated performance requirements and move to the next step in the acquisition life cycle 
within cost and schedule while integrating the COTS, GOTS, or other proposed NDI. This 
review is held instead of a PDR and CDR, but addresses the same technical rigor of these 
reviews. The MCSC SETR organizational structure as well as planning, execution, and 
close out processes are similar to those discussed in the SETR process overview. The 
MCSC SETR Handbook has a detailed appendix for each SETR event that includes an 
event overview, entrance criteria, review elements, agenda, exit criteria, and evaluation 
criteria. This resource acts as a job aid and future reference as the MCSC-related programs 
are executing events to help ensure consistency across the organization, which is a key 
document strength. 
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Figure 5. SETR Tailoring In Relation to Acquisition Life Cycle. Source: DON 

MCSC (2014, 3). 

SPAWARINST 5401.4, which is titled Systems Engineering Policy, establishes 
the systems engineering policy for the command. Within this instruction, it directs that 
SPAWAR 5.0 will conduct SETRs to support “the delivery of an integrated, 
interoperable, and tested capability to the Elect” (DON SPAWAR 2016b, 2). As stated in 
the SETR overview, this will be documented by programs in the SEP; but in addition 
SPAWAR mandates that it must be completed and submitted to SPAWAR 5.0 within 
three months of program initiation. Prom a SPAWAR perspective, this is the first 
instruction that includes guidance on the SETR implementation. 

SPAWARINST 5400.3A, which is titled Systems Engineering Technical Review 
Policy, establishes the policy and responsibilities associated with the SETR 
implementation for SPAWAR. Key roles are held by the SPAWAR CHENG and the 
respective PM when conducting the SETR. The associated enclosure, which is the 
Systems Engineering Technical Review Organization Standard Process Handbook, 
provides the core of the implementation guidelines to the SYSCOM. 

SPAWAR’s Systems Engineering Technical Review Organization Standard 
Process Handbook goes into the most detailed tailoring options of all the SYSCOM 
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documentation. The handbook provides a SETR implementation for each of the following 
models that can be implemented within the SYSCOM: 

• Model 1: Hardware Intensive Program (traditional SETR) 

• Model 2: Defense Unique Software Intensive Program 

• Model 3: Incrementally Eielded Software Intensive Program 

• Model 4: Accelerated Acquisition Program 

Model 1 aligns to what was diseussed in the SETR overview. However, Eigure 6 shows 
an image to keep context as each of the additional models are reviewed. 
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Eigure 6. Model 1 Hardware Intensive Program. Source: DON SPAWAR 

(2016c, 18). 


Model 2 comes into play when there is a defense unique software intensive program. The 
hardware has to be capable of being separated from the software builds to allow more 
tailored approaeh on the SETR side for software. However, the combined software and 
hardware builds are deployed on a consistent combined schedule. This approach would 
be used when several software builds are necessary for a hardware build that is being 
deployed. This is the first introduction for SPAWAR of the Build Technical Review 
(BTR) and Eielding Technical Review (ETR) events, which again would be used for the 
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software while the hardware follows a more typical SETR implementation (DON 
SPAWAR 2016c). These two new reviews are designed to “assess systems engineering 
rigor for requirements decomposition, design, development, and testing to allow greater 
flexibility and response to evolving technologies” per the handbook (22). In this case, the 
BTR would be supporting the incremental build and the FTR would be supporting the 
software side of the combined hardware/software fielding decision that a program would 
have for a traditional SETR event. Figure 7 shows this in an image to get a better idea of 
how the risk reduction occurs from a software perspective with the multiple BTR events 
during a normal timeframe to gain a better understanding of the technical maturity of the 
defense software and readiness to move to the next stage of the program’s life cycle. 
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Figure 7. Model 2 Defense Unique Software Intensive Program. 

Source: DON SPAWAR (2016c, 19). 


The next model in the SPAWAR Systems Engineering Technical Review Organization 
Standard Process Handbook takes the biggest departure from the traditional SETR 
implementation with incrementally fielded software intensive program model. Model 3 
also makes use of the BTR and FTR concepts. Due to the more pure software nature of 
the program coupled with the new SETR events instead of the traditional SETR events. 
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the program can make more rapid software deployments to the warfighter as shown in 
Figure 8 by the incremental builds depicted in each SETR “row.” The SW would be built 
agnostic to the specific HW, but compliant with HW standards. 
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Figure 8. Model 3 Incrementally Fielded Software Intensive Program. 

Source: DON SPAWAR (2016c, 23). 


The final model shown in Figure 9 and provided by the SPAWAR Systems Engineering 
Technical Review Organization Standard Process Handbook is the most rapid SETR 
model depicted. The handbook points out that this model should be used when “schedule 
dominates over cost and technical risk considerations” and should be used when 
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“technological surprises by a potential adversary necessitates a higher-risk acquisition 
program” (24). So even though it is the most rapid, it is rather specific in terms of when it 
should be leveraged. This model introduces an additional event in the handbook called 
the SETR-Lite Engineering Review (SER), which, as it sounds, is a highly tailored and 
focused review to support a specific program decision; but still contains the rigor of other 
SETR events. In an accelerated acquisition program as depicted in Eigure 9, a SER is 
executed at a minimum twice. However, the SER is a useful review for projects that may 
not meet the threshold for a traditional or even tailored SETR implementation, so a SER 
can be executed to meet the intent of an independent technical review as often as needed 
in the project life cycle. 
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Eigure 9. Model 4 Accelerated Acquisition Program. 

Source: DON SPAWAR (2016c, 25). 


E. INDUSTRY 


IEEE Standard 15288.2-2014, which is the Standard for Technical Reviews and 
Audits on Defense Programs, details the traditional SETR implementation; however, 
much like described in the process tailoring section for IEEE Standard 15288, there is a 
section up front in the document that speaks to process tailoring and expectations. Unlike 
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the DOD or DON documentation, the IEEE Standard 15288.2-2014 includes a solid 
baseline definition list, which includes items such as the following: 

acceptability criteria: A documented set of characteristics of a program’s 
work products that if satisfied, forms a sufficient basis for judging each 
product’s content to be acceptable to support a successful review or audit. 

entry criteria: Artifacts and other review or audit elements that must be 
completed before the review or audit can be conducted. 

exit criteria: Review or audit elements that must be assessed, completed, 
and action items closed before successful completion of the technical 
review or audit can be declared. 

technical reviews: A series of systems engineering activities conducted at 
logical transition points in a system life cycle, by which the progress of a 
program is assessed relative to its technical requirements using a mutually 
agreed-upon set of criteria. (IEEE Computer Society 2014, 4-5) 

The IEEE Standard 15288.2-2014 goes through each traditional SETR event to explain 
what makes up a successful event. Starting off, each section explains the acceptability 
criteria for the event with regards to each product (e.g., system specification) that will be 
reviewed at the event. The standard moves on to detail what is required for event 
preparation broken down by the responsible person and their associated actions. The 
actual execution of the event is then explained along with what will be reviewed in each 
part of the SETR event itself. Einally, this section of the standard speaks to what is 
needed to successfully close out this SETR event from each person and for what actions 
are they responsible. While the NAVSEA 05H Technical Review Manual contained great 
detail about planning to closure of a SETR event, this IEEE Standard 15288.2-2014 goes 
step by step for each traditional SETR event including what it takes from planning all the 
way to closure of each individual event, which is similar to the level of detail in the 
MCSC SETR Handbook. 

The Software Engineering Institute (SEI) released a technical note on Agile 

Methods: Selected DOD Management and Acquisition Concerns in October 2011, which 

frames the use of agile as a mechanism to address the “disconnect between the 

warfighter/demand tempo and the acquisition/contracting tempo” (Software Engineering 

Institute [SEI] 2011, 3). The technical note addresses topics such as contract execution, 
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contract monitoring, and sustainment when a program is leveraging agile development 
methodology. Of note from a SETR perspective, the technical note describes two 
potential solutions for dealing with more traditional technical reviews to include specific 
advantages and disadvantages of each. The first option is translation of agile products 
into the traditional review events. Whether this translation is done by the vendor 
themselves or another support entity (e.g.. Systems Engineering and Technical Assistance 
contractor), the program, as pointed out by the SEI technical note, is incurring a cost to 
translate the agile material into traditional documents, which may outweigh the benefit 
agile provides. The second option the SEI technical note provides is to execute more 
iterative reviews, or mini events, that build up to a more traditional SETR event. This 
second option is similar to the tailored NAVAIR’s RBR or SPAWAR’s BTR. 

F. SUMMARY 

The DOD and DON directives and instructions lay the system engineering 
foundation for the naval SYSCOMs guidance regarding SETR implementation. The 
following are the key attributes from each of the higher level DOD and DON documents 
reviewed in this chapter: 

• The Defense Acquisition System (DODD 5000.01): Eays the foundation 
with a standard definition of what systems engineering is within the DOD. 

• Operations of the Defense Acquisition System (DODI 5000.02): Includes 
requirement for PDR and CDR within DOD level instruction along with 
criteria for when applicable. 

• Business Systems Requirements and Acquisition (DODI 5000.75): 
Supersedes DODI 5000.02 for defense business systems; includes 
reference to technical assessments that must be addressed in the 
implementation plan. 

• Deputy Assistant Secretary of Defense for Systems Engineering (DODI 
5134.16): Initial SEP requirement introduced at DOD level. 

• ASN RD&A Memo, SETR Process for Naval Acquisition Programs, 13 
June 08: Directs SETR implementation across naval SYSCOMs. 

• Naval SYSCOM Systems Engineering Policy (SPAWARINST 5000.1): 
Specifies the minimum SETR elements (e.g., SETR chair, agenda, 
schedule, expected results) across the naval SYSCOMs. 
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• DON Implementation and Operation of the Defense Acquisition System 
and the Joint Capabilities Integration and Development System 
(SECNAVINST 5000.2E): Introduction of “IT Box” concept. Underscores 
the importance of being able to tailor the program’s SETR implementation 
to keep pace with the increased design to deployment timeline that “IT 
Box” enables. 

Each SYSCOM has varying guidance regarding SETR to include tailoring. The 
following are the key attributes from each of the SYSCOM documents reviewed in this 
chapter: 

• NAVSEA OSH Technical Review Manual, Version 2.0, 18 December 
2009: Provides most comprehensive traditional SETR guidance, but does 
have reminders for tailoring mentioned throughout. Appendix includes 
detailed descriptions of entrance and exit criteria for each SETR event, 
different documentation, architectural views, and scheduling details. 

• NAVAIR Systems Engineering Technical Review Process (NAVAIRINST 

4355.19E): Includes specific definition for tailoring SETR 

implementation. Introduces a tailored event—RBR—for agile software 
development. 

• NAVAIR Adapting Acquisition to Agile Software Development: A How- 
To Guide, Version 2.0, 19 March 2014: Provides execution level details on 
the RBR. 

• MCSC SETR Handbook (SIAT-HDBK-001): Encourages tailoring 
opportunities and even introduces the NIR, which is a tailored event. 
Detailed appendix for each SETR event that includes an event overview, 
entrance criteria, review elements, agenda, exit criteria, and evaluation 
criteria 

• SPAWAR Systems Engineering Policy (SPAWARINST 5401.4): 
Mandates SEP, which is where SETR implementation is documented, 
must be completed and submitted to SPAWAR 5.0 within three months of 
program initiation. 

• SPAWAR Systems Engineering Technical Review Policy (SPAWARINST 
5400.3A): Establishes key SETR roles within SPAWAR. 

• SPAWAR Systems Engineering Technical Review Organizational 
Standard Process Handbook. Provides a tailored SETR implementation 
for each DOD software acquisition models, which is the most extensive 
tailoring guidance in all of the SYSCOM documents. 
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The following list includes key attributes from the industry documents reviewed 
in this chapter: 

• IEEE Technical Reviews and Audits for Defense standard: provides solid 
definitions (e.g., exit criteria, entrance criteria). With each event, the 
standard provides preparation instruction and other information to ensure 
the event is successful. 

• SEI Agile Methods: Selected DOD Management and Acquisition 
Concerns: addresses topics such as contract execution, contract 
monitoring, and sustainment when a program is leveraging agile 
development methodology. Describes two potential solutions for dealing 
with more traditional technical reviews 

The following chapter presents details on the method for collecting the data as 
well as presentation of the data gathered from the survey. 
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IV. PRESENTATION OE SURVEY DATA 


A. INTRODUCTION 

To address the research questions, a survey was leveraged to gather data points 
from SMEs executing SETRs to understand how SETRs are being tailored, what impacts 
the SETR has on future acquisition steps, and what hurdles programs are currently facing. 
These survey responses are in the context of the current policies, instructions, and other 
job aides that are in place across the DOD, DON, and naval SYSCOMs. Upon conclusion 
of the analysis, recommendations will be provided based on lessons learned on how to 
increase likelihood of a program ROI for a tailored SETR implementation as it relates to 
naval software acquisitions. This chapter will provide details on the method for collecting 
the data as well as presentation of the survey data gathered. Appendix B provides the 
comprehensive survey data set. Chapter V provides the research analysis and discussion 
tying the policy review chapter to the data from the survey to address the research 
questions. 

As suggested in Eowler’s Survey Research Methods, one of the first steps in 
initiating a survey is choosing the appropriate sample frame to understand those who will 
be selected to participate in the survey as the target population (Eowler 2014). Eor this 
specific research, the survey was distributed across the engineering workforce at 
SPAWAR Systems Center Atlantic (SSC EANT) as well as shared with the other naval 
SYSCOMs where feasible. The target audience within these organizations was SMEs 
who have expertise organizing, leading, and/or participating in a program SETR event. 
SSC EANT engineering technical workforce is made up of about 3,000 government 
employees (Eigure 10). The survey was focused within this engineering technical 
workforce to a more granular audience of SETR SMEs. SSC EANT supports other DON 
customers (Eigure 11) as a working capital funded organization, which provided a unique 
opportunity to gain insight into other DON program SETR implementations through this 
survey population. The top five sponsors include four of the naval SYSCOMs— 
SPAWAR, MCSC, NAVSEA, and NAVAIR. As the SSC EANT technical workforce 

supports programs across these other SYSCOMs, the survey generated results that 
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provided insight across DON programs more completely than a survey of a single 
SYSCOM. 
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The online SurveyMonkey tool was used to distribute and collect the data from 26 
April through 17 May 2017. Overall, the survey had 48 respondents. The survey did not 
require a user to provide identifying information, other than what organization they most 
closely associated with in order to understand the breadth of organizations that were 
sampled in the survey. The Naval Postgraduate School Institutional Review Board 
determined that the research is not designed to collect information about individuals and 
therefore is not human subject research. The survey questions, which are provided in 
Table 4, did collect a mix of objective and subjective data points. The objective data 
points consisted of those that are factually based, such as size of program, events tailored, 
and area of program adjusted (Fowler 2014). The subjective data points relate to a SMEs 
experience such as what events were most valuable. These questions were worded to 
reduce error such as personalizing the response by ensuring that the program was the 
acting noun not the person. 
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Table 4. Survey Questions and Potential Responses 


No. 

Survey Questions 

Question Type 

Response Options 

1 

Which organization are 
you most closely 
associated with? 

Multiple Choice 
(Single Answer) 

Department of Defense 

Department of Navy 

Marine Corps Systems Command 

Naval Air Systems Command 

Naval Eacilities Engineering Command 

Naval Sea Systems Command 

Naval Supply Systems Command 

Space and Naval Warfare Systems Command 
Other 

2 

What is the size of the 
program that used the 

SETR process? 

Multiple Choice 
(Single Answer) 

ACATI 

ACAT II 

ACAT III 

Other (please specify) 

3 

What type of software 
acquisition model did this 
program leverage? 

Multiple Choice 
(Single Answer) 

Commercial off-the-shelf 

Government off-the-shelf 

New Development 

Other (please specify) 

4 

Did the program tailor the 
system engineering 
technical review process 
to support your program? 

Multiple Choice 
(Single Answer) 

Yes 

No 

5 

If applicable, which part 
did you tailor? 

Multiple Choice 
(Multiple Answer) 

Review Order 

Reviews Included 

Entrance/Exit Criteria 

Not Applicable 

Other (please specify) 

6 

What reviews did the 
program include in the 
SETR process? 

Multiple Choice 
(Multiple Answer) 

Build Technical Review 

Capability Build Review—Initial 

Capability Build Review—Design 

Capability Build Review—Readiness 

Critical Design Review 

Fielding Technical Review 

Functional Configuration Audit 

Integration Readiness Review 

Operational Test Readiness Review 

Physical Configuration Audit 

Preliminary Design Review 

Production Readiness Review 

Release Backlog Review 

SETR-Lite Engineering Review 

Software Specification Review 

System Functional Review 

System Requirements Review 

System Verification Review 

Test Readiness Review 

Other (please specify) 
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No. 

Survey Questions 

Question Type 

Response Options 

7 

Which review events did 
the program find most 
valuable to assessing the 
technical maturity of the 
program? 

Multiple Choice 
(Multiple Answer) 

Build Technical Review 

Capability Build Review—Initial 

Capability Build Review—Design 

Capability Build Review—Readiness 

Critical Design Review 

Eielding Technical Review 

Eunctional Configuration Audit 

Integration Readiness Review 

Operational Test Readiness Review 

Physical Configuration Audit 

Preliminary Design Review 

Production Readiness Review 

Release Backlog Review 

SETR-Lite Engineering Review 

Software Specification Review 

System Functional Review 

System Requirements Review 

System Verification Review 

Test Readiness Review 

Other (please specify) 

8 

What area of the program 
was adjusted based on the 
outcome of the SETR? 

Multiple Choice 
(Multiple Answer) 

Cost 

Schedule 

Technical Design 

No change 

Other (please specify) 

9 

By what percentage of the 
cost, schedule, and/or 
technical design was the 
program adjusted, if 
applicable? 

Multiple Choice 
(Single Answer) 

0-20% 

21-40% 

41-60% 

61-80% 

81-100% 

Not Applicable 

10 

Did any of the following 
options limit or create 
obstacles if the program 
tailored the SETR 
implementation? 

Multiple Choice 
(Multiple Answer) 

Internal Organization 

External Organization 

Policy 

No limitation 

Not applicable 

Other (please specify) 

11 

Did the program include 
any metrics to assess the 
return on investment from 
the reviews? 

Multiple Choice 
(Single Answer) 

Yes (please specify). No 

12 

Did the program have 
consistent leadership 
throughout a complete 
SETR cycle? 

Multiple Choice 
(Single Answer) 

Yes, No 

13 

Please provide any 
additional feedback 
regarding SETR 
implementations. 

Eree Text 

N/A 


39 













B. SURVEY DATA 


(1) Question 1 

Figure 12 provides the results to the initial question—’’Which organization are 
you most closely associated with?” The intent of this question was to understand from a 
sampling perspective which organizations were covered or not during the survey period. 
SPAWAR was the SYSCOM that the majority of participants most closely associated 
themselves with, followed by the DON. 


Q1 Which organization are you most closely 
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Answered: 48 Skipped: 0 
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Figure 12. Question 1 Survey Results 
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(2) Question 2 

Figure 13 provides the results to the second question—’’What is the size of the 
program that used the SETR process?” This question was included to understand, from a 
sampling perspective what mixture of AC AT program types was covered in the survey. 
Most survey participants were involved in ACAT I programs. Seven of the nineteen 
responses in the “other” category specified that the program size was an abbreviated 
acquisition program (AAP). Per SECNAVINST 5000.2E, AAP are programs at a high 
level that do not require operational test and evaluation as documented by the Operational 
Testing Agency (OTA) and meet a specific dollar threshold per the instruction. The 
complete set of responses provided in the free text field—’’Other (please specify)”—are 
included in Appendix B. 


What is the size of the program that used 
the SETR process? 

Answered: 48 Skipped: 0 
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Eigure 13. Question 2 Survey Results 
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(3) Question 3 

Figure 14 provides the results to the next question—’’What type of software 
acquisition model did this program leverage?” This question was incorporated into the 
survey to help answer the research question regarding if there were any specific tailoring 
concerns based on the type of acquisition model the program leveraged. Most of the 
participants acquired COTS based on the responses provided. Ten of the thirteen “other” 
responses included some combination of the original response options provided (e.g., 
COTS and GOTS, all). The complete set of responses provided in the free text field— 
’’Other (please specify)”—are included in Appendix B. 


Q3 What type of software acquisition model 
did this program leverage? 

Answered: 48 Skipped. 0 
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Figure 14. Question 3 Survey Results 
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(4) Question 4 

Figure 15 provides the results to the fourth question—’’Did the program tailor the 
SETR process to support your program?” This question begins to get at the heart of the 
thesis regarding whether programs are leveraging the tailoring allowed by the current 
policies in places across the DOD, DON, and their respective SYSCOMs. Based on the 
participant’s responses, programs do appear to be taking advantage of the various 
tailoring opportunities described in the current policies as over 80% responded yes the 
program did tailor the SETR implementation. 
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Eigure 15. Question 4 Survey Results 

(5) Question 5 

Eigure 16 provides the results to the fifth question, which is a follow on to the 
previous question—”If applicable, which part did you tailor?” The intent behind this 
question is to understand how the programs tailored by using this multi-select question. A 
program could respond with not applicable or multiple other options. Tailoring reviews 
included and entrance/exit criteria were the most common responses from survey 
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participants. Of the eight “other” responses, three of them ineluded skipping required 
documentation. The complete set of responses provided in the free text field—’’Other 
(please specify)”—are ineluded in Appendix B. 


If applicable, which part did you tailor? 

Answered: 44 Skipped: 4 


Review Order 
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Figure 16. Question 5 Survey Results 


(6) Question 6 

Figure 17 provides the results to the next question—’’What reviews did the 
program include in the SETR process?” This question was to baseline which reviews 
from the potential options provided in the SYSCOM policy documentation to include the 
known tailored reviews (e.g., NAVAIR RBR, MCSC NIR, SPAWAR SER). The most 
eommonly included reviews were PDR, CDR, SRR, and TRR. The specific responses in 
the “other” category listed tailored events such as combining SRR and PDR into a single 
event, so there was not a eonsistently repeated answer. The complete set of responses 
provided in the free text field—’’Other (please specify)”—are included in Appendix B. 
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Figure 17. Question 6 Survey Results 
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(7) Question 7 

Figure 18 provides the results to the seventh question—’’Which review events did 
the program find most valuable to assessing the technical maturity of the program?” This 
question was included to address the research question focused on the critical reviews of 
the SETR implementation that program managers and/or lead engineers should include in 
a tailored implementation. The most common answer to this question was PDR, CDR, 
and TRR, which was closely followed by SRR. The complete set of responses provided 
in the free text field—’’Other (please specify)”—are included in Appendix B. 
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Figure 18. Question 7 Survey Results 
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(8) Question 8 

Figure 19 provides the results to the next question—’’What area of the program 
was adjusted based on the outcome of the SETR?” The outcome of a SETR 
implementation overall should be to ensure that the program is on track, or adjust based 
on the risk found in the review to ensure follow on phases of the acquisition process do 
not increase in risk. This question was intended to understand whether based on the 
SETR events, the program had to adjust the cost, schedule, technical design, or some 
other part of the program prior to proceeding to the next acquisition phase. Schedule and 
technical design were the top two responses indicating areas the program adjusted based 
on the outcome of the SETR. The complete set of responses provided in the free text 
field—’’Other (please specify)”—are included in Appendix B. 


Q8 What area of the program was adjusted 
based on the outcome of the SETR? 

Ar.swered: 48 Skipped: 0 



Eigure 19. Question 8 Survey Results 
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(9) Question 9 

Figure 20 provides the results to the ninth question—”By what percentage of the 
cost, schedule, and/or technical design was the program adjusted, if applicable?” If the 
answer to the previous question indicated there was an adjustment made, then this 
question sought to understand how significantly the program adjusted after the SETR 
review. If an adjustment was made to the program based on the SETR outcome, based on 
the responses to this survey, the change would most likely be in the 0-20% range, which 
was the response for over half the participants. 


Q9 By what percentage of the cost, 
schedule, and/or technical design was the 
program adjusted, if applicable? 

Anrwered:48 Skipped 0 
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Eigure 20. Question 9 Survey Results 
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(10) Question 10 

Figure 21 provides the results to the next question—’’Did any of the following 
options limit or create obstacles if the program tailored the SETR implementation?” This 
survey question aligns to a research question focused on addressing the obstacles (e.g., 
policy, timelines, and maturity) and risks faced by naval programs that have attempted to 
tailor the SETR process. More specifically do these obstacles vary based on the size of 
the program (e.g., ACAT I vs non program of record)? The most common response to 
this survey question was that the external organization created limits and/or obstacles to 
tailoring the SETR implementation. The complete set of responses provided in the free 
text field—’’Other (please specify)”—are included in Appendix B. 


Q10 Did any of the following options limit or 
create obstacles if the program tailored the 
SETR implementation? 

Answered; 48 Shipped: 0 
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Eigure 21. Question 10 Survey Results 
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(11) Question 11 

Figure 22 provides the results to the eleventh question—’’Did the program include 
any metrics to assess the return on investment from the reviews?” One area that was not 
addressed heavily in any of the government policy documentation was metrics, so this 
question was included to see if lack of mandatory inclusion carried through to the reviews 
themselves. The SPAWAR Systems Engineering Technical Review Organization 
Standard Process Handbook did include some specific metric examples as noted in Table 
2. SETR implementations are iterative events throughout different phases of the program 
life cycle as mentioned at various points in this thesis. Metrics are one way to baseline 
the outcome of an event and, as follow on events occur, understand if improvements are 
occurring. Based on the responses to the survey, the policy seems to be indicative of the 
actual execution of the SETR events as almost 80% did not include any metrics to 
understand if there was a ROI from the reviews. The complete set of responses provided 
in the free text field—”Yes (please specify)”—are included in Appendix B. 


Q I I Did the program include any metrics to 
assess the return on investment from the 

reviews? 

Answered: 47 Sklppe i! 



Yes (please 
specify) 
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Eigure 22. Question 11 Survey Results 
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(12) Question 12 

Figure 23 provides the results to the twelfth question—’’Did the program have 
consistent leadership throughout a complete SETR cycle?” Leadership as highlighted 
throughout each of the DOD, DON, and SYSCOM documentation primarily through the 
program PM is critical to ensuring successful SETR implementation, so this question was 
included to see if the programs included in the survey had consistent leadership through a 
complete SETR cycle. The response from this question appears fairly split with almost 
half having consistent leadership throughout a complete SETR cycle and the other not 
having consistent leadership. 


Q12 Did the program have consistent 
leadership throughout a complete SETR 

cycle? 

Answei^d: 47 Skipped: ' 
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Eigure 23. Question 12 Survey Results 
(13) Question 13 

The final question was an open ended question—’’Please provide any additional 
feedback regarding SETR implementations.” The responses varied from general feedback 
regarding SETR to how tailoring was implemented and included feedback on the survey 
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itself. The complete set of responses provided in this free text field are included in 
Appendix B. This chapter provided details on the method for collecting the data as well 
as the data gathered from the survey. Chapter V will address the research questions 
through survey data analysis and details from the policy and standard review to include 
recommendations based on the responses from the final open ended question. 

C. SUMMARY 

The survey was distributed across the engineering workforce SSC LANT as well 
as shared with the naval SYSCOMs where feasible. The target audience within these 
organizations was the SMEs who have expertise organizing, leading, and/or participating 
in a SETR event for a program. The online SurveyMonkey tool was used to distribute and 
collect the data from 26 April through 17 May 2017. The survey did not require a user to 
provide identifying information other than what organization they most closely associated 
with, in order to understand the breadth of SYSCOMs that were sampled in the survey. 
The survey questions did collect a mix of objective and subjective data points. Overall, 
the survey had 48 respondents, which included varying ACAT program sizes, software 
acquisition models, and degrees of tailoring the process. The following chapter provides 
the research analysis, which ties the literature review section to the data from the survey 
and addresses the research questions. 
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V. RESEARCH ANALYSIS AND DISCUSSION 


A. INTRODUCTION 


This chapter provides the research analysis, which ties the literature review 
chapter to the survey data to address the research questions. The literature review focused 
on the DOD, DON, naval SYSCOM policies, and industry documentation providing 
direction and guidance related to the process, entrance criteria, exit criteria, leadership 
involvement, tailoring, and all other applicable aspect of the SETR process. Leveraging a 
tailored SETR implementation provides the necessary structured engineering framework 
while keeping up with a dynamic software development environment to meet increasing 
need for enhanced capability delivered to the warfighter in a shorter timeframe. 
Specifically, this research addresses how the SETR implementation can be tailored to 
most efficiently leverage resources and minimize schedule impact while addressing the 
acquisition, technical, and policy/legal requirements within naval software acquisitions. 
In order to answer this question, the following subsidiary questions will also be 
addressed. 


1. What are the obstacles (e.g., policy, timelines, and maturity) and risks faced 
by naval programs that have attempted to tailor the SETR process? Do these 
obstacles vary based on the size of the program (e.g., ACATI vs non 
program of record)? 

2. What are the critical elements of the current system engineering technical 
review process (e.g., entrance/exit criteria, order of reviews, risk 
assessment) that program managers and/or lead engineers should include in 
a tailored implementation? What incentives or return on investment do these 
critical elements provide? 

3. Are there any considerations/differences that should be accounted for when 
tailoring the process to support various acquisition models (e.g., COTS, new 
development, GOTS)? 

4. What can engineers and program managers of future naval acquisitions 
learn from other programs that have attempted to tailor? 
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B. RESEARCH QUESTION ANALYSIS 
I. Primary Research Question 

The primary research question is “how can the SETR implementation be tailored 
to most efficiently leverage resources and minimize schedule impact while addressing the 
acquisition, technical, and policy/legal requirements within naval software acquisitions?” 
Frank Kendall, then Acting Undersecretary of Defense for Acquisition, Technology, and 
Logistics (AT&L), addressed the question of what is the optimal program structure, 
which parallels to the optimal SETR structure. The optimal SETR structure is dependent 
upon many variables which, IEEE 15288 Annex A provides a good reference for 
consideration. 

The answer to the question is either: (A) There is none, or (B) There are an 
infinite number. There is no one best way to structure a program. Every 
program has its own best structure, and that structure is dependent on all 
the many variables that contribute to program success or failure... .There is 
no one solution. What I’m looking for fundamentally is the evidence that 
the program’s leaders have thought carefully about all of the factors that 
I’ve mentioned— and many others. I look for that evidence in the nature 
of the product the program is acquiring and in the structure the program’s 
leaders have chosen to use. The thinking (and the supporting data) that 
went into determining that specific and often unique structure is what I 
expect to see in an acquisition strategy, and it is what I expect our leaders 
to be able to explain when they present their program plans. (Kendall 
2012, 23) 

Over 80% of the survey respondents from across various DON organizations indicated, 
as shown in Figure 15, that tailoring was occurring within programs. In a tailored SETR 
implementation, the programs find the most ROI by tailoring entrance criteria, exit 
criteria, and/or which specific reviews are included based on the responses to survey 
question five as shown in Figure 16. Programs are primarily adjusting the schedule and/or 
technical design based on the outcome of the SETR events as shown in Figure 18. The 
percentage of adjustment could be upwards of 60% based on the response to question 
nine as shown in Figure 19. Over 50% of the survey respondents indicated their programs 
adjusted the cost, schedule, and/or technical design by up to 20%. However, fifth 
question’s response structure leads to some uncertainty in this result as the answer 
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included zero as part of the option, so some respondents could have been choosing this 
option—”0-20%”—to indicate no change. When the response to question 8, regarding the 
program areas that were adjusted based on SETR outeome, are compared to the 
percentage of adjustment (question 9), it appears that four of the responses did select the 
option—”0-20%”—to indicate no ehange as shown in Table 5. Table 6 provides the 
original response count/percentages and the revised to account for this issue. The option 
“0-20%” still was most prevalent response with the adjustment made for this question 
response structure issue. 


Table 5. Adjusted program areas (Question 8) Compared to Pereent Adjusted 

(Question 9) 



0-20% 

(1) 

21-40% 

(2) 

41.60% 

(3) 

61.80% 

(4) 

81-100% NOT APPLICABLE 

(5) ■ (6) 

TOTAL 

- Q8; Cost (A) 

78.57% 

11 

0.00% 

0 

21 43% 

3 

0.00% 

0 

0.00% 

0 

0.00% 

0 

30.43% 

14 

- Q8: 

62.50% 

16.67% 

20.83% 

0.00% 

0.00% 

0.00% 

52.17% 

Schedule (8) 

15 

4 

5 

0 

0 

0 

24 

- Q8: 

68.00% 

8.00% 

16.00% 

0.00% 

0.00% 

8.00% 

54.35% 

Technical 

Design (C) 

17 

2 

4 

0 

0 

2 

25 

- Q8: No 

36.36% 

0.00% 

0.00% 

0.00% 

0.00% 

63.64% 

23.91% 

change(D) 

4 

0 

0 

0 

0 

7 

11 

- Total 

Respondents 

27 

4 

6 

0 

0 

9 

46 


Table 6. Revised Response Calculations 


Question 9: By what percentage of the cost, schedule, and/or technical design was 
the program adjusted, if applicable? 

Answer Options 

Response 

Percent 

Response 

Count 

Revised 

Response 

Percent 

Revised 

Response 

Count 

0-20% 

56.3% 

27 

47.9% 

23 

21-40% 

8.3% 

4 

8.3% 

4 

41-60% 

12.5% 

6 

12.5% 

6 

61-80% 

0.0% 

0 

0.0% 

0 

81-100% 

0.0% 

0 

0.0% 

0 

Not Applicable 

22.9% 

11 

31.3% 

15 

answered question 

48 


skipped question 

0 
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When comparing the respondent’s response of what was tailored in the program 
(question 5) to the percentage the program cost, schedule, and/or technical design 
(question 9) was adjusted as shown in Table 7, the programs that tailored the SETR 
implementation focusing the tailoring on which reviews to include seemed to have the 
largest return based on the percent the program was adjusted. In addition, fewer SETR 
events should indicate less time and resources spent preparing for unneeded reviews. 
Additional statistical analysis should be performed in this area. 


Table 7. SETR Areas Tailored (Question 5) Compared to Percent Adjusted 

(Question 9) 


- 

0-20% 

(1) 

21-40% 

(2) 

41-60% 

(3) 

61-80% 

(4) 

81-100% NOT APPLICABLE 

(5) ■ (6) 

TOTAL 

Q5: Review 

16.67% 

16.67% 

33.33% 

0.00% 

0.00% 

33.33% 

13.95% 

Order (A) 

1 

1 

2 

0 

0 

2 

6 

Q5: Reviews 

54.84% 

9.68% 

19.35% 

0.00% 

0.00% 

16.13% 

72.09% 

Included (B) 

17 

3 

6 

0 

0 

5 

31 

- Q5; 

65.52% 

3.45% 

13.79% 

0.00% 

0.00% 

17.24% 

67.44% 

Entrance/Exit 
Criteria (C) 

19 

1 

4 

0 

0 

5 

29 

- Q5: Not 

57.14% 

0.00% 

0.00% 

0.00% 

0.00% 

42.86% 

16.28% 

Applicable 

(D) 

4 

0 

0 

0 

0 

3 

7 

- Total 

Respondents 

24 

3 

6 

0 

0 

10 
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Metrics to track the ROI throughout SETR implementation is an item that is 
difficult to address based on respondent’s answers to survey question eleven as shown in 
Eigure 22. Only nine respondents indicated that any metrics were included in the SETR 
process. In reviewing the additional details provided in the open ended response area for 
this question, most metrics were number of items completed (e.g., action item tracking). 
A higher level of action items completed is not necessarily a metric of success as it could 
indicate that the program should not have entered the event in the first place because the 
program was not sufficiently ready. Euture research would be valuable in this area as the 
policy and other documentation did not add much direction and/or examples. Erom a 
literature review perspective, SPAWAR Systems Engineering Technical Review 
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Organization Standard Process Handbook was the only document that addressed metrics 
beyond a brief mention. 

2. Subsidiary Question 1 

The first subsidiary question is “what are the obstacles (e.g., policy, timelines, and 
maturity) and risks faced by naval programs that have attempted to tailor the SETR 
process? Do these obstacles vary based on the size of the program (e.g., ACAT I vs non 
program of record)?” More than 50% of the survey respondents indicated that “External 
Organizations” created the most obstacles (question 10) as shown in Eigure 21. 
Interestingly, when reviewing the “other” free form responses, five of the nine responses 
could have also been categorized as “External Organization” as shown in Table 8. When 
looking at the responses to question 10 through the lens of the program size (e.g., ACAT 
I, ACAT II), it does not seem to affect the response as “External Organization” is 
consistently the response more than 50% of the time as shown in Table 9. Euture research 
should include statistical analysis in this area. 


Table 8. Categorization of “Other” Survey Responses for Question 10 


Question 10: Other (please specify) 

Categorization 

1 

it depends. For instance, if the PM/LSE uses a SETR event that senior 
ieadership/MDA is not famiiiar with, then it may resuit in additionai 
exchanges between the PM and senior ieadership/MDA to educate both 
sides untii both sides agree on how it wiii be addressed and executed. 

Externa i 
Organization 

2 

Every Navy ‘Heavy-weight’ that controis money in the budget thinks they 
can become a computer guru by buying the iatest and greatest product that 
a saiesperson can taik them into, and don’t seem to reaiize that aii 
requirements of ANY systems shouid be known PRiOR TO any purchase. 
Money to spend doesn’t make a manager an expert, and they wiii be 
rotated out of that job in (usuaiiy) three years (or iess) when someone eise 
wiii come in and make decisions aii over again, if you don’t know how a 
present system runs (to fuifiii aii the iaws, poiicies, required today), then 
how can you say that you are an expert quaiified to repiace that system, if 
you force the conversion on partiai knowiedge then the risk of faiiure 
increases and aii members of that service suffer and Congress is unhappy. 
Why? Because they controiied the budget and had the authority but did not 
have sufficient knowiedge, or training, or experience, to make the right caii 
knowing the risks. 

Externa i 
Organization 


Personnei- The organization was not staffed with the appropriate number 
of peopie. 

internai 

3 

Skiiiset -The work force was not trained for the positions they heid. 

Organization 
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Question 10: Other (please specify) 

Categorization 

4 

PMW 240, as a portfolio of smaller programs, is built around the idea that 
the SETR process needs to be tailored for each, individual program; 
leadership is also aware that defense business systems, especially 
unmodified COTS implementations, are unique among DOD acquisitions, 
which necessitates modifications to the SETR process 

Internal 

Organization 

5 

The Program Manager did not and currently does not care for engineers. 
They “get in his way.” 

Maybe that’s why his program is failing...? 

Internal 

Organization 

6 

Actual SETR events were determined by MCSC leadership external to the 
Program Office. 

External 

Organization 

7 

There is very few tailoring guidelines. This leads to significant 
disagreement about what tailoring is acceptable. This problem exists at all 
levels. Take architecture for example: the DoDAF specifications are vague 
on tailoring, the Navy Architecture Development Guide does not directly 
address tailoring, and the SPAWAR architecture review guidelines (used by 
contractors to conduct reviews) make no provision for tailoring. Until 
tailoring is addressed directly at all levels from policy to review guidelines, 
attempting to tailor will lead to significant friction, frustration, and delays. 

External 

Organization 

8 

The external organizations (user and customer) were not involved in the 
review. Obstacles arose in the approval of a design that the user had never 
seen and making design decisions driving cost and schedule that the 
customer did not have input to change. 

External 

Organization 


Table 9. Size of Program (Question 2) Compared to Obstacles (Question 10) 



INTERNAL 
ORGANIZATION - 
(1) 

EXTERNAL 
ORGANIZATION - 
(2) 

POLICY 

(3) 

NO 

LIMITATION - 
(4) 

NOT 

APPLICABLE - 
(5) 

OTHER 

(PLEASE 

SPECIFY) 

(6) 

TOTAL 

- Q2: ACAT1 

(A) 


29.41% 

5 


58.82% 

10 

17.65% 

3 

11.76% 

2 

11.76% 

2 

17.65% 

3 

Responses 

86.21% 

25 

- Q2: ACAT II 

(B) 


0.00% 

0 


100.00% 

1 

0.00% 

0 

0.00% 

0 

0.00% 

0 

0.00% 

0 

Responses 

3.45% 

1 

_ Q2: ACAT III 

(C) 


18.18% 

2 


54.55% 

6 

0.00% 

0 

9.09% 

1 

18.18% 

2 

18.18% 

2 

Responses 

44.83% 

13 

Total 


7 


17 

3 

3 

4 

5 

29 


Respondents 


3. Subsidiary Question 2 

The second subsidiary question is “what are the critical elements of the current 
system engineering technical review process (e.g., entrance/exit criteria, order of reviews, 
risk assessment) that program managers and/or lead engineers should include in a tailored 
implementation? What incentives or return on investment do these critical elements 
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provide?” Based on the survey responses when a program tailors its SETR 
implementation, the primary focus is on the reviews included and what entrance and/or 
exit criteria is included. Interestingly, in the open-ended responses to survey question 5 
the respondents agreed that tailoring the acquisition documentation was another key area 
of focus. When determining which events were most valuable (question 7) to include in 
the tailored SETR implementation, PDR, CDR, and TRR ranked 20% points or greater 
than any of the other survey options. This research question would benefit in future 
research from additional statistical analysis to better tie the return on investment to the 
type of tailoring that was implemented. 

4. Subsidiary Question 3 

The third subsidiary question is “are there any considerations/differences that 
should be accounted for when tailoring the process to support various acquisition models 
(e.g., COTS, new development, GOTS)?” Based on comparison of the responses in 
questions 3 and 4, the software acquisition model does not affect whether the program 
tailored the SETR implementation or not as shown in Table 10. Eurther, Table 11 
indicates that the software acquisition model does not affect a program’s focus on SETR 
tailoring from the three primary elements: the reviews included, entrance criteria, and exit 
criteria. 


Table 10. Software Acquisition Model (Question 3) Compared to if SETR 

Tailored (Question 4) 

- YES(1) - NO (2) - TOTAL 

- Q3: 80.95% 19.05% 60.00% 

Commercial 17 4 21 

off the Shelf 

(A) 

- Q3: 

Governmeht 
off the Shelf 

(B) 

Q3: New 
Development 

(C) 

- Total 29 6 35 

Respondents 


75.00% 

3 


90.00% 

9 


25.00% 

1 


11.43% 

4 


10 . 00 % 

1 


28.57% 

10 
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Table 11. Software Acquisition Model (Question 3) Compared to SETR Areas 

Tailored (Question 5) 



REVIEW 
ORDER (1) 

REVIEWS 

INCLUDED 

(2) 

ENTRANCE/EXIT 
CRITERIA (3) 

NOT 

APPLICABLE - 
(4) 

OTHER 
(PLEASE 
SPECIFY) (5) 

TOTAL 

Q3: 

26.32% 

73.68% 

63.16% 


21.05% 

21.05% 

121.88% 

Commercial 
off the Shelf 
(A) 

5 

14 

12 


4 

4 

Responses 

39 

Q3: 

0.00% 

75.00% 

50.00% 


25.00% 

0.00% 

18.75% 

Government 
off the Shelf 
(B) 

0 

3 

2 


1 

0 

Responses 

6 

Q3: New 

11.11% 

88.89% 

77.78% 


0.00% 

11.11% 

53.13% 

Development 

(C) 

1 

8 

7 


0 

1 

Responses 

17 

Total 

Respondents 

6 

25 

21 


5 

5 

32 


When comparing the software acquisition model (question 3) to the specific 


reviews included (question 6) as shown in Table 12, the COTS and new development 
models included most consistently the PDR and CDR as SETR events, which is not 
surprising these are the two reviews given DODI 5000.02 includes the requirement for 
these two reviews along with specific criteria on when the requirement applies. 
Expectations for these two events are better understood and engrained in the culture due 
to this guidance being at the DOD level. The comparison data on the GOTS software 
acquisition model is more evenly distributed across the review types as only four 
respondents fell into this software acquisition model, so more data would be needed to 
verify if this holds true for a GOTS software acquisition model. 
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Table 12. Software Acquisition Model (Question 3) Compared to Specific 

Reviews Included (Question 6) 


- 

03; COMMERCIAL OFF THE 

03: GOVERKWENT OFF THE 

03: NEV/ 

TOTAL 


SHELF(A) 


SHELF (B; 


DEVELOPMENT (C) 


BUM 


6COO% 


2Q.X% 

20.X% 

14.71% 

Tecflrlcai 

(1) 


3 


1 

1 

5 

CapaMity 


5CX% 


C.X% 

5C.X% 

5.88% 

BUM RevISM 
- inmai i2) 


1 


0 

1 

2 

CapaMity 


66.67% 


O.X% 

33.33% 

8.82% 

BUM Review 
-Design (3) 


2 


0 

1 

3 

CapaMity 

BlIm Review 


66 67% 

2 


33.33% 

1 

O.X% 

0 

8.82% 

3 

- Reaaress 

W 







cnvcai 


5417% 


8.33% 

37.50% 

70.59% 

Design 

Revie* (5) 


13 


2 

9 

24 

Hewng 


5C.0C% 


33.33% 

16.67% 

17.65% 

Tecfirlcai 

Review (6) 


3 


2 

1 

6 

Funcaonal 


5C0C% 


2C.X% 

3Q.X% 

29.41% 

Canflguranan 

Aij(a(7) 


c 


2 

3 

10 

inegranon 


44 44% 


C.X% 

55.56% 

26.47% 

ReadoesE 
Review (8) 


4 


0 

5 

9 

Operaionai 


5C0C% 


25.M% 

25.X% 

35.29% 

Test 

ReadresE 

Review 


6 


3 

3 

12 

Pl'yslcal 


54 55% 


18.18% 

27.27% 

32.35% 

Conflguratlon 
Audi (10) 


6 


2 

3 

11 

Preandnary 


5i17% 


8.70% 

39.13% 

67.65% 

Deagn 

Review (11) 


12 


2 

9 

23 

Prodxdon 


54 55% 


O.X% 

45 45% 

32.35% 

ReadresE 
Review (12) 


6 


0 

5 

11 

Re^ee 


25 «% 


O.X% 

75.m% 

11.76% 

BacKKig 

Review (13) 


1 


0 

3 

4 

SETR^J^e 


4C.X% 


4C.X% 

2C.X% 

14.71% 

En^neerlng 
Review (1 a; 


2 


2 

1 

5 

Sonttm 


2657% 


14.29% 

57.14% 

20.59% 

SpecillcaKn 
Review (IS) 


2 


1 

4 

T 

SyElem 


41.18% 


17.65% 

41.18% 

50.00% 

FuniMonal 
Review (16) 


7 


3 

7 

17 

System 


6C.OO% 


8.X% 

32.X% 

73.53% 

Requremerls 
Review (17) 


IS 


2 

8 

25 

System 


42.86% 


7.14% 

50.X% 

41.18% 

Verfflcatlor 
Review (18) 


6 


1 

7 

14 

Test 


55 56% 


11.11% 

33.33% 

79.41% 

ReadresE 
Review (19) 


15 


3 

9 

27 

(Over (please 


70. X% 


1C.X% 

20.X% 

29.41% 

Epeciry)(20) 


7 


1 

2 

10 



Reeponeee 


Recponeee 

Rdepon&ee 


Total 


2C 


4 

10 

34 
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5. 


Subsidiary Question 4 


The fourth subsidiary question is “what can engineers and program managers of 
future naval acquisitions learn from other programs that have attempted to tailor?” The 
final open ended survey question as well as the “other” free form response areas in 
various questions provided the most valuable, direct insight to address this question. One 
survey respondent of the open-ended question included the following: 

While SETR is a great tool to assess where a program is in the time of the 
event. We are normally limited in time and resources on how deep to 
conduct the review. We are also limited to the information that the 
programs provide. 


I had an instance where the performers provided all the necessary 
information, everything appeared on schedule for delivery and the SETR 
went by smoothly. I later found out that the performers were 11 months 
behind schedule, but this was not apparent with the provided 
documentation. 


While SETR is a good tool, it is often cumbersome to the programs to 

support this effort while still performing their daily tasks. 

Time and resource constraints are key reasons that tailoring is typically 
considered. However, as noted in the respondent’s comment, the review and associated 
artifacts being evaluated should be carefully selected to hit the critical areas without 
adding unnecessary overhead on the program, avoid causing the SETR event to become a 
“check-the-box” event, and/or miss risks such as being eleven months behind schedule. 
Based on the survey results, key concepts that are most impactful to a tailored SETR 
implementation’s success are as follows: 

• engage leadership 

• focus on preparation 

• maintain early and often communication 

• educate stakeholders 

Eeadership engagement came up repeatedly through-out the survey responses. As 


Secretary Kendall states in his description of the optimal program structure, the 
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individual’s thinking is a critical piece, which requires an engaged leader (Kendall 2012, 
23). The program’s SETR leadership should seek out other program lessons learned 
including potential review templates when determining how they are going to structure 
the implementation. Similarly, once the program has initiated SETR events, the program 
SETR leadership should willingly look for opportunities to share with other programs to 
ensure best practices continue to evolve. One respondent highlighted a leadership 
challenge with respect to engaging and understanding the variables to address in order to 
avoid events becoming “a ‘check-the-box’ sort of event if the program is required to 
complete every single SETR review, or even to address every single ‘bullet point’ for a 
given review.. .SETR reviews are absolutely necessary, but forcing programs to complete 
reviews that are extraneous given their specific situation leads to increased costs, 
schedule, etc.” As noted in the literature review, the PM is the ultimate recipient of the 
SETR report that captures the confidence level of the baseline (DON SPAWAR 2009, 
1012). The manner in which the PM addresses the report sends a direct message to the 
rest of the program. By choosing to take an active role in ensuring action items or other 
issues are addressed, the PM lets the rest of the program office know that SETR 
implementation is not a “check-the-box” sort of event. 

SETR events require focused preparation, which is probably best described in the 
following SYSCOM documents that provide detailed preparation information (e.g., 
scheduling, review elements) for each review: 

• NAVSEA OSH Technical Review Manual, Version 2.0, 18 December 
2009: Appendix includes detailed descriptions of entrance and exit criteria 
for each SETR event, different documentation, architectural views, and 
scheduling details. 

• NAVAIR Adapting Acquisition to Agile Software Development: A How- 
To Guide, Version 2.0, 19 March 2014: Provides execution level details on 
the RBR. 

• MCSC SETR Handbook (SIAT-HDBK-001): Detailed appendix for each 
SETR event that includes an event overview, entrance criteria, review 
elements, agenda, exit criteria, and evaluation criteria. 

As noted by a survey respondent, the SETR events “typically take a block of 
preparatory time and resolution time to address REAs;” however, “the benefit is that 
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multiple SETR events ensure documentation is being updated as the program matures 
versus the documents becoming static and outdated.” The program must balance focused 
preparation with ensuring the program is looking at what events will provide the most 
ROI for the program such as the PDR or CDR prior to jumping in and just “checking the 
box.” 

Early and often communication was a concept that developed when reviewing the 
survey responses as well as literature review to enable management of expectations and 
reduce risk in executing a tailored SETR event. As a respondent noted “managing 
expectations & frequent sync sessions are critical to success. Echelon 2 & Echelon 3 have 
different interpretations of what is expected regarding the level of completeness of each 
SETR.” Waiting until the program has fully vetted internally the tailored SETR 
implementation without reaching out to external stakeholders increases the time and 
chances of obstacles being put in the program’s path. SETR should be looked at as a way 
to engage users, internal and external stakeholders to deal with risks before they become 
issues in an open transparent manner, which requires early and often communication. 

Similarly, educating stakeholders is a key piece to ensuring success with a 
tailored SETR implementation. The SETR implementation whether tailored or not must 
be documented in the program’s SEP, which is generally signed by individuals outside of 
the program office. Due to this, if the program determines that a tailored SETR 
implementation is best then the program must educate not only the participants of the 
event, but also external stakeholder(s) that have to approve the SEP. This could take 
significant amount of time as one respondent noted, “if the PM/ESE uses a SETR event 
that senior leadership/MDA is not familiar with, then it may result in additional 
exchanges between the PM and senior leadership/MDA to educate both sides until both 
sides agree on how it will be addressed and executed.” As noted in the literature review 
section, there are a few tailored SETR implementation examples documented in DOD, 
DON, or naval SYSCOM documents to lean on during this discussion, which one 
respondent noted “until tailoring is addressed directly at all levels from policy to review 
guidelines, attempting to tailor will lead to significant friction, frustration, and delays” as 
the program has to overcome the culture of what people find acceptable. Without 
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supportive program leadership, overcoming this culture could prove too difficult for some 
SETR leadership teams leading to the “check-the-box” mentality. 

C. SUMMARY 

Leveraging a tailored SETR implementation provides the necessary structured 
engineering framework while keeping up with a dynamic software development 
environment to meet increasing need for enhanced capability delivered to the warfighter 
in a shorter timeframe. Over 80% of the survey respondents from across various DON 
organizations indicated, as shown in Eigure 15, that tailoring was occurring within 
programs to address program specific needs such as aggressive schedule and leveraging 
an agile software development models. The tailored SETR events are directly impacting 
the program’s next phase through technical design adjustments and other key program 
variables. The reviews included specifically the PDR and CDR, as well as entrance and 
exit criteria, as the aspects that programs find provide a ROT This is the case for both 
COTS and new development software acquisition models. Eactors external to the 
organization continue to be the primary obstacle no matter the program size determining 
the ability to successfully tailor and implement the SETR process based on the responses 
to the direct survey questions and open ended questions. Euture naval software 
acquisition programs should engage leadership, focus on preparation, maintain early and 
often communication, and educate stakeholders to improve ROI of the tailored SETR 
implementation. The following chapter presents the conclusions from the research and 
suggestions for follow on research. 
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VI. CONCLUSION 


A. INTRODUCTION 

According to the Naval SYSCOM Systems Engineering Policy (2009), the SETR 
process should be an event-driven process that consists of one or more programmatically 
independent reviews with defined entrance and exit criteria. The SETR reviews assess the 
technical health, requirements accuracy, design maturity, testing effectiveness, and 
sustainment support over the program life cycle. The jointly signed SYSCOM policy 
states that the program SETR events, along with the event entrance and exit criteria, are 
documented in the SEP which is signed by the program MDA or other designated 
authority based on the program ACAT level. The SYSCOM policy notes that event 
closure normally occurs only after the exit criteria has been met, but the SETR TRB 
chair must concur with the identified action items along with any plan of action and 
milestones and/or mitigations. According to the policy, the SETR output ultimately 
informs the PM if the program is technically ready to move on to the next phase of the 
acquisition process. 

Prom a policy perspective, the DOD and DON directives and instructions lay the 
systems engineering foundation for the naval SYSCOM guidance regarding SETR 
implementation. In addition, the Navy has emphasized the need to deliver capability 
versus systems, and acquisition is impacted by this capability vision requiring innovative 
application of SETR for the SoS or platform level efforts, which traditional SETR is not 
well structured to support. Eeveraging a tailored SETR implementation provides the 
necessary structured engineering framework while keeping up with a dynamic software 
development environment to meet increasing need for enhanced capability delivered to 
the warfighter in a shorter timeframe. More than 80% of the survey respondents from 
across various DON organizations indicated, as shown in Pigure 15, that tailoring was 
occurring within programs to address program specific needs such as aggressive schedule 
and leveraging an agile software development model. The tailored SETR events are 
directly impacting the program’s next phase through technical design adjustments and 

other key program variables. The reviews included specifically the PDR and CDR, as 
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well as entrance and exit criteria, as the aspects that programs find provide a ROI when 
tailoring the SETR implementation. This is the case for both COTS and new development 
software acquisition models. Factors external to the organization continue to be the 
primary obstacle no matter the program size determining the ability to successfully tailor 
and implement the SETR process. 

B. RECOMMENDATIONS 

Future naval software acquisition programs should engage leadership, focus on 
preparation, maintain early and often communication, and educate stakeholders to 
improve ROI of the tailored SETR implementation. As Secretary Kendall states in his 
description of the optimal program structure, the individual’s thinking is a critical piece, 
which requires engaged leaders (Kendall 2012, 23). Seeking out other program lessons 
learned prior to determining how to tailor the program’s SETR implementation will 
reduce the likelihood of tailored implementations that provide no ROI on the program’s 
SETR implementation. 

Similarly, once the program has initiated SETR events the program SETR 
leadership should willingly look for opportunities to share with other programs to ensure 
best practices continue to evolve. The program must balance focused preparation with 
ensuring the program is including reviews in the tailored implementation that will 
provide the most ROI for the program (such as the CDR) prior to jumping in and just 
“checking the box.” 

Educating stakeholders is a key piece to ensuring success with a tailored SETR 
implementation. Additionally, waiting until the program has fully vetted internally the 
tailored SETR implementation without reaching out to communicate with external 
stakeholders increases the time and chances of obstacles being put in the program’s path. 
SETR should be looked at as a way to engage users, internal and external stakeholders to 
deal with risks before they become issues in an open transparent manner, which requires 
early and often communication. The SETR implementation, whether tailored or not, must 
be documented in the program’s SEP, which is generally signed by individuals outside of 
the program office. Due to this, if the program determines that a tailored SETR 
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implementation is best, then the program must educate not only the participants of the 
event, but also external stakeholders that have to approve the SEP. This could take 
significant amount of time. As noted in the literature review section, there are a few 
tailored SETR implementation examples documented in DOD, DON, or naval SYSCOM 
documents to lean on during this discussion. Without supportive program leadership, 
overcoming this culture could prove too difficult for some SETR leadership teams 
leading to the “check-the-box” mentality. 

Erom a policy prospective, tailoring needs to be addressed directly at all levels to 
avoid significant friction, frustration, and delays. Ideally, the SYSCOM guidance or 
supplemental documentation should provide examples of successfully tailoring and 
lessons learned that future naval programs can use as a starting point to avoid having to 
learn the lessons the hard way. Additionally, the SYSCOM level documentation should 
address minimal level metrics to objectively measure SETR implementation ROI to 
ensure the events are not “check-the-box” events, which only serve to drain already 
constrained resources within the program. 

C. FUTURE RESEARCH 

As the DON continues to focus priorities around delivering capabilities versus 
individuals systems, additional research to improve the SETR implementation across the 
department is critical, especially to stay current with industry software best practices, 
which is leading to more rapid software deployments. Then-Secretary Gates framed the 
challenge in a September 2008 speech. “Our conventional modernization programs see a 
99% solution in years. Stability and counterinsurgency—the wars we are in—require a 
75% solution in months. The challenge is whether in our bureaucracy and in our minds 
these two different paradigms can be made to coexist” (SEI 2011, xi). Warfighters always 
benefit from finding ways to reduce the bureaucracy and deliver enhanced capabilities in 
a shorter timeframe addressing the dynamic threats they face daily. The alternative to 
allowing bureaucracy to triumph, from a DON perspective, has potential consequences 
such as loss of life that should motivate every individual involved in the SETR 
implementation to achieve the greatest ROI in the shortest amount of time. 
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More than 80% of survey respondents stated that the program did not include 
metrics to assess the ROI of the SETR reviews. From a literature review perspective, 
SPAWAR Systems Engineering Technical Review Organization Standard Process 
Handbook was the only document that addressed metrics beyond a brief mention. 
Research identifying metrics to assess ROI would not only help future programs from a 
lesson learned perspective, but also could help programs executing the SETR event 
understand how to adjust future reviews gaining more value on what can be delivered to 
the warfighter. Another related research area is the average SETR investment from the 
program as it relates to the average ROI (e.g., cost savings, schedule savings). This 
research would benefit holistically from additional statistical analysis to better tie the 
return on investment to the type of tailoring that was implemented. 

Beyond metrics, an area that developed during this research due to DODI 5000.75 
being published was the difference in a SETR event for a business system versus a 
national security system. DODI 5000.75 supersedes DODI 5000.02 for all business 
system acquisition, which directs alignment “to commercial best practices and will 
minimize the need for customization of commercial products to the maximum extent 
possible” (DOD 2017, 4). This should lead to more accepted use of SETR tailoring, 
which will provide opportunities to further this research. 

D. SUMMARY 

This research provides tailored SETR implementation recommendations based on 
lessons learned on how to increase positive ROI for a naval software acquisitions 
program. The recommendations will assist program leadership in making better decisions 
on where to allocate software engineering resources within the schedule and funding 
constraints. While the thesis is focused on the DON, the recommendations are applicable 
to SETR implementations for software acquisitions programs across the broader DOD. In 
addition, three future research topics are provided based on areas that came up in this 
effort that could be expanded upon. Tailored SETR implementation has not been heavily 
researched specifically as it relates to software acquisition programs, so there is a 
significant opportunity to positively impact the product the warfighter receives. 
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APPENDIX A. LEADING INDICATORS 


The following tables show leading indieators in relation to IEEE Standard 15288. 


Table 4 - 

LEADING INDICATORS APPLICATION 

PER 

ISO/IEC 15288 


Requirements T rends 

System Definition Change 

Backlog Trend 

Intertece Trends 

Requirements Validation 

Trends 

Requirements 

Verification Trends 

Work Product Approval 

Trends 

Review Action Closure 

Trends 

Risk Exposure Trends 

Risk T reatment T re nds 

Technology Maturity 

Trends 

Technical Measurement 

Trends 

Systems Engineering 

Staffing & Skills Trends 

Process Compliance 

Trends 

Tes t Co m pi ete ne ss 

Trends 

Facility and Equipment 

Availability Trends 

Defect/Error Trends 

System Affordability 

Trends 

Architecture Trends 

Scheduleand Cost 

Pressure 

6.3 Protect Processes 

















6.3.1 Protect Ptannino Process 




















6.3.1.3.a Defirre the protect 




















6.3.1.3.b Plan the protect resources 












X 



X 




X 

6.3.1.3.C Plan the project technical and qualHv 

maneoement 






X 

X 









X 




6.3.1.3.d Activate the protect 




















6.3.2 Protect Assessment and Control Process 




















6.3.2.3JI Assess the protect 






X 

X 





X 

X 


X 

X 



X 

6.3.2.3.b ContrcH the protect 






X 

X 





X 

X 


X 

X 



X 

6.3.2.3.C Close the protect 




















6.3.3 Decision Manaqement Process 




















6.3.3.3.a Plan and deflrte decisions 










X 







X 



6.3.3.3.b Analyze the decision information 










X 







X 



6.3.3.3.C Track the decision 










X 







X 



6.3.4 Risk Manaoement Process 




















6.3.4.3.a Plan Risk Manaoement 




















6.3.4.3.b Manaoe Risk Profile 




















6.3.4.3.C Analyze Risks 








X 












6.3.4.3.d Treat Risks 








X 

X 











6.3.4.3e Monitor Risks 








X 

X 











6.3.4.3.f Evtiuate Risk Manaqement Process 








X 
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6.3.5 Configuration Management Process 




















6.3.5.3.a Plan configuration manaoen>ent 




















6.3.5.3.b Perform configuration manaoement 


X 
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Table 4 - LEADING INDICATORS APPLICATION 

PER 

ISO/IEC 15288 




Req u ire me nts T ren d s 

System Definition Change 

Backlog Trend 

Internee Trends 

Requirements Validation 

Trends 

Requirements 

Verification Trends 

Work Product Approval 

Trends 

Review Action Closure 

Trends 

Risk Exposure Trends 

R is k Treatment Tre rKl s 

Technology Maturity 

Trends 

Technical Measurement 

Trends 

Systems Engineering 

Staffing & Skills Trends 

Process Compliance 

Trends 

Test Completeiress 

Trends 

Facility and Equipment 

Availability Trends 

Defect/ Error T rends 

System Affordability 

Trends 

Architecture Trends 

Scheduleand Cost 

Pressure 

6.3.6 Informatfon Manaoement Process 




















6.3.6.3.a Plan information martaoement 




















6.16.3.b Perform Information manaqement 
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6.4 Technical Processes 




















6.4.1 Stakeholder Requirements Definition 
Process 




















6.4.1.3.a Elicit Stakeholder Requirements 
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6.4.1.3.b Define Stakeholder Reaulrements 

X 
















X 



6.4.1.3.C Analyze and Maintain Stakeholder 
Requirements 

X 
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X 







X 






X 



6.42 Requirements Analysis Process 




















6.4.2.3.a Define System Requirements 

X 
















X 



6.42.3.b Analyze and Maintain System 
Requirements 

X 

X 


X 

X 






X 






X 



6.4.3 Architectural Desiqn Process 




















6.4.3.3.a Define Architecture 



X 







X 







X 

X 


6.4.3.3.b Analyze and Evaluate Architecture 



X 







X 

X 






X 

X 


6.4.3.3.C Document and Maintain Architecture 


X 

X 
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6.4.4 Implenrentation Process 




















6.4.4.3.a Plan Implementation 




















6.4.4.3.b Perform implementation 











X 









6.4.5 Inteoratlon Process 




















6.45.3.a Plan Integration 














X 






6.45.3.b Perform integration 











X 
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6.4.6 Verification Process 




















6.4.6.3.a Plan verification 





X 
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Table 4 - LEADING INDICATORS APPLICATION PER ISO/IEC 15288 


Requirements Trends 

System Definition Change 

Backlog Trend 

Interfoce Trends 

Requirements Validation 
Trends 

Requirements 
Verification Trends 

Work Product Approval 
Trends 

Review Action Closure 
Trends 

Risk Exposure Trends 

Risk Treatment Trends 

Technology Maturity 
Trends 

Technical Measurement 
Trends 

Systems Engineering 
Staffing & Skills Trends 

Process Compliance 
Trends 

Test Completeness 
Trends 

Facility and Equipment 

AvaiiabilityTrends 

Defect/Error Trends 

System Affordability 

Trends 

Ar ch it ectu re T ren d 5 

Schedule and Cost 

Pressure 

6.4.6.3.b Perform verification 





X 









X 






6.48 Validation Process 




















6.4.8.3.a Plan validation 




X 










X 






6.48.3.b Perform validation 




X 










X 






6.410 Maintenance Process 




















6.410.3.a Plan maintenance 










X 










6.4.10.3.b Perform maintenance 










X 
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APPENDIX B. COMPLETE SURVEY DATA 


Question 1: Which organization are you most closely associated with? 

Answer Options 

Response 

Percent 

Response 

Count 

Department of Defense 

4.2% 

2 

Department of Navy 

27.1% 

13 

Marine Corps Systems Command 

6.3% 

3 

Naval Air Systems Command 

2.1% 

1 

Naval Facilities Engineering Command 

0.0% 

0 

Naval Sea Systems Command 

2.1% 

1 

Naval Supply Systems Command 

2.1% 

1 

Space and Naval Warfare Systems Command 

54.2% 

26 

Other 

2.1% 

1 

answered question 

48 

skipped question 

0 


Question 2: What is the size of the program that used the SETR process? 

Answer Options 

Response 

Percent 

Response 

Count 

ACATI 

35.4% 

17 

ACAT II 

2.1% 

1 

ACAT III 

22.9% 

11 

Other (please specify) 

39.6% 

19 

answered question 

48 

skipped question 

0 


Question 2: Other (please specify) 

1 

MCSC/PEO ES acquisition programs range from ACAT I down to AAPs 

2 

Not ACAT. Program is less than $3 million. 

3 

Non-POR 

4 

Non-Program of record 

5 

Project 

6 

Most Eegacy systems don’t use SETR 

7 

One POR consisting of multiple abbreviated acquisition programs (AAP); ACAT 

III equivalent overall 

8 

AAP and an ACAT III program 

9 

ACAT 4 and Non-ACAT programs 
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10 

Abbreviated Acquisition Program (most recent) 

11 

Smaller programs that support ACAT I-III programs 

12 

abbreviated acquisition systems 

13 

AAP 

14 

AAP 

15 

ACAT IV (T) 

16 

Not a program of record; an R&D program pre-milestone B equivalent to ACAT I 

17 

R&D Demonstration Program 

18 

Demonstration Program vice Acquisition program. 

19 

All ACAT level programs within PEO C4I go through some SETR events. 


Question 3: What type of software acquisition model did this program leverage? 

Answer Options 

Response 

Percent 

Response 

Count 

Commercial off-the-shelf 

43.8% 

21 

Government off-the-shelf 

8.3% 

4 

New Development 

20.8% 

10 

Other (please specify) 

27.1% 

13 

answered question 

48 

skipped question 

0 


Question 3: Other (please specify) 

1 

Maximum use of COTS, with some defined use of GOTS. 

2 

Depends on the acquisition program 

3 

Non Program of Record (Non-POR) 

4 

Rapid IT Acquisition—COTS 

5 

In house Development years ago, Being maintained but not suitable to SETR, 
being phased out if they can ever find a COT fit (but is not advisable, at all, in my 
book). 

6 

A combination of COTS, GOTS, and new development followed by an Integration 
activity. 

7 

both COTS/GOTS 

8 

Mix of COTS (Office-type products), GOTS (Eor architecture), and New Plug-in 
Development 

9 

Maintenance of existing SPAWAR-developed software plus some new 
development. 

10 

Some new development with large amount of integration with COTS and GOTS 

11 

All of the above. 

12 

We deal with all of these activities. 
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I don’t understand this question. 


13 


On one effort we used the IT Box process, on others we tailored to meet our needs 
for the acquisition process. 


Question 4: Did the program tailor the system engineering technical review 
process? 

Answer Options 

Response 

Percent 

Response 

Count 

Yes 

81.3% 

39 

No 

18.8% 

9 

answered question 

48 

skipped question 

0 


Question 5: If applicable, which part did you tailor? 

Answer Options 

Response 

Percent 

Response 

Count 

Review Order 

13.6% 

6 

Reviews Included 

70.5% 

31 

Entrance/Exit Criteria 

65.9% 

29 

Not Applicable 

15.9% 

7 

Other (please specify) 

18.2% 

8 

answered question 

44 

skipped question 

4 


Question 5: Other (please specify) 

1 

All of the above. Each acquisition program is unique and tailors it program based 
on many factors. 

2 

Those entrance and exit criteria that does not apply to non-ACAT or Non-POR. 

3 

Required Acquisition Documentation 

4 

Also combined multiple events into one (i.e. SRR/SER, instead of two separate 
reviews/events); held “light” versions of various events (e.g., PDR, CDR) after 
major requirements and design overhauls 


AAP tailored the entrance/exit criteria for all SETR events 

5 

ACAT III has tailored all SETR events to the point where the effectiveness of 
events have been compromised. 

6 

literally everything. The existing SPA WAR SETR process is wholly overbearing 
and useless to line engineers. Had to rescope our review of contractor deliverables 
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to a very simple Requirements Review + Initial Design Review (aka PDR) + Final 
Design Review (aka CDR) + Testing Review (TRR + Test Results all rolled into 
one). 


7 

Seope of doeumentation and reviews. The existing software was developed before 
the SETR process applied; thus, not all required documentation now required exists. 
Because the program is in sustainment, funding to retroactively develop the 
documentation is not available. 

8 

Applicable documentation. For my demonstration program the systems engineering 
process is focused on requirements definition with less focus on life cycle 
engineering as that is not prudent unless the program transitions to a full scale 

ACAT program. 


Question 6: What reviews did the program include in the SETR process? 

Answer Options 

Response 

Percent 

Response 

Count 

Build Technical Review 

17.0% 

8 

Capability Build Review—Initial 

4.3% 

2 

Capability Build Review—Design 

6.4% 

3 

Capability Build Review—Readiness 

6.4% 

3 

Critical Design Review 

68.1% 

32 

Fielding Technical Review 

17.0% 

8 

Functional Configuration Audit 

27.7% 

13 

Integration Readiness Review 

21.3% 

10 

Operational Test Readiness Review 

36.2% 

17 

Physical Configuration Audit 

29.8% 

14 

Preliminary Design Review 

68.1% 

32 

Production Readiness Review 

29.8% 

14 

Release Backlog Review 

10.6% 

5 

SETR-Fite Engineering Review 

14.9% 

7 

Software Specification Review 

19.1% 

9 

System Functional Review 

46.8% 

22 

System Requirements Review 

70.2% 

33 

System Verification Review 

36.2% 

17 

Test Readiness Review 

76.6% 

36 

Other (please specify) 

36.2% 

17 

answered question 

47 

skipped question 

1 


Question 6: Other (please specify) 

1 

Flight Readiness Review 
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2 

What SETR reference is being used for the list provided? Some programs create 
their own unique technical reviews. There are other SETRs that are not in the list 
provided. 

3 

System Requirements Review and System Eunctional Review were combined. 

4 

ATO, ATP, CRD, EOT&T, IBR, lEA, PD&E, REI, REP, UAT 

5 

Navy HR Pers/Pay systems have so many unique requirements that ongoing 
maintenance will be cheaper than conversion and the cost of new licenses, 
customization, upgrades, new licenses, re-customizations to the upgrades and that 
repetitive cycle over and over through time. 

6 

TRR was followed by independent component and integration testing. 

7 

Our initial Engineering Master Plan called out combining 2 SETR events into 
each review, with a total of 6 reviews. 


AAP tailored entrance/exit criteria for SRR, PDR, CDR, PRR, EGA, SVR, PCA, 
TRR. 

8 

ACAT III focus is on speed to capability and is using SETR-Eite Engineering 
Reviews to accelerate process. 

9 

The SRR and PDR was combined into a single event 

10 

Combined SRR/SER 

11 

Non-Development Item Integration Review (NIR) in lieu of PDR/CDR for 
integration of COTS products. 

12 

USMC-specific Non-Developmental Item (NDI) Integration Review (NIR) which 
is comparable to a CDR but for NDI/COTS items. See MCSC Technical Review 
Handbook. 

13 

Other review were conducted, but not while I was on the project 

14 

NIR also included. 

15 

OTRR 

16 

Many of the elements of the non-implemented reviews are captured via quarterly 
program review events. Regular course check on technical progress vice milestone 
reviews. Also, contractors handle a lot of the systems engineering milestones at 
their level. Some reviews are held with government participation and insight, but 
not a government approval panel. 

17 

We also in what I believe is called an In-process review (I dont recall off hand but 
it is a mini review of the effort at a given time—between other SETR reviews). 

Not sure if that qualifies but it acts like a PMR but focused on engineering efforts. 


Question 7: Which review events did the program find most valuable to assessing 
the technical maturity of the program? 

Answer Options 

Response 

Percent 

Response 

Count 

Build Technical Review 

4.2% 

2 

Capability Build Review—Initial 

4.2% 

2 

Capability Build Review—Design 

8.3% 

4 

Capability Build Review—Readiness 

6.3% 

3 
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Critical Design Review 

56.3% 

27 

Fielding Technical Review 

4.2% 

2 

Functional Configuration Audit 

4.2% 

2 

Integration Readiness Review 

12.5% 

6 

Operational Test Readiness Review 

12.5% 

6 

Physieal Configuration Audit 

8.3% 

4 

Preliminary Design Review 

39.6% 

19 

Produetion Readiness Review 

12.5% 

6 

Release Baeklog Review 

4.2% 

2 

SETR-Lite Engineering Review 

6.3% 

3 

Software Specification Review 

10.4% 

5 

System Eunetional Review 

12.5% 

6 

System Requirements Review 

29.2% 

14 

System Verifieation Review 

16.7% 

8 

Test Readiness Review 

39.6% 

19 

Other (please speeify) 

12.5% 

6 

answered question 

48 

skipped question 

0 


Question 7: Other (please specify) 

1 

Teehnieal maturity is eontinuously assessed along the acquisition life eyele. 

2 

NSIPS was estimated to require no more than 15 to 20 to install initially, but 
required 90% ehange to implement. It has been required to upgrade with new 
licenses, re-eustomization, and requires a 30 day period to complete end to end 
testing of changes. Eegacy can change in 1 to 2 days including testing all 
requirements. Navy Eegaey HR data is on mainframe; not ever haeked. New 
systems want to use “eloud” teehnology, whieh ean be haeked. Navy members 
should never be in a setting that is known to be easy to hack. Veterans/Service 
Members should know their HR data is safe, not subjeet to any haeking; veterans 
and eurrent members deserve to have their data loeked up and proteeted to the best 
of our ability to do that ! ! ! 

3 

Our Program is still Pre-Milestone B (ATP) and in REP release stage. We have only 
been through CDR and SRR. Our next seheduled SETR is not until Q2 EY19. 

4 

NIR 

5 

Did not find any valuable. Only time consuming. 

6 

Quarterly Program Reviews. 


Question 8: What area of the program was adjusted based on the outcome of the 
SETR? 

Answer Options 

Response 

Percent 

Response 

Count 
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Cost 

29.2% 

14 

Schedule 

50.0% 

24 

Technical Design 

52.1% 

25 

No change 

22.9% 

11 

Other (please specify) 

20.8% 

10 

answered question 

48 

skipped question 

0 


Question 8: Other (please specify) 

1 

Various areas may be adjusted based on the outcome of the SETR. However, the 
outcomes are based on PM decisions since SETR close-out reports are from the 
program-independent SETR chair and to the PM. The SETR close-out report will 
include recommendations for the PM to consider. 

2 

Depending on what was going to be available or ready for delivery, what will be 
included in the release package. 

3 

Engineers want computers and computer science codified to engineering standards, 
and while some of those concepts apply, some don’t. Computer science, to have as 
a tool for improving office systems and business systems is not the same as 
(similar, but different) computers for robotic type of situations, or warfare types of 
situations. No amount of training (in engineering steps) can replace a well 
qualified IS/IT/ADP top-of-the line specialist. Engineers will try to prove they can 
over and over, but that relies on the concept of saying one size fits all (which is 
true in some cases) which does not expand on the qualified IT Specialist’s ability 
to adopt/adept/fits right, the needs of a business to a particular situation, i.e.. 

Military HR PERS/PAY are unique but must be a fit to serve each service to fulfill 
the needs of all applicable laws Congress can pass. 

4 

Contract modifications occurred as result of the SETR Design reviews discovering 
that there was lack of interoperability with fielded capabilities. Additionally, the 
Design Reviews produced reprioritization of the interfaces. 

5 

System Requirements Specification 


Eor ACAT III—no change 

6 

for AAP—Cost, Schedule and technical design 

7 

SETR had no impact on what the PM wanted to do. He did what he wanted to do 
regardless of what the output of the SETR said. 

8 

Typically design was refined—not changed. Significant design changes occurred 
later during implementation that caused substantial delay. The NIR was treated by 
leadership as a check-in-the-box event rather than a substantial design review— 
perhaps the design issues could have been avoided if treated otherwise. 

9 

Cost more / schedule ended up much longer. 

10 

Reviews were so tailored in terms of entry/exit criteria, that the reviews were 
driven to ensure passing. This meant all risks associated were deemed “acceptable” 
and basically transformed into issues. 
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Question 9: By what percentage of the cost, schedule, and/or technical design was 
the program adjusted, if applicable? 

Answer Options 

Response 

Percent 

Response 

Count 

0-20% 

56.3% 

27 

21-40% 

8.3% 

4 

41-60% 

12.5% 

6 

61-80% 

0.0% 

0 

81-100% 

0.0% 

0 

Not Applicable 

22.9% 

11 

answered question 

48 

skipped question 

0 


Question 10: Did any of the following options limit or create obstacles if the 
program tailored the SETR implementation? 

Answer Options 

Response 

Percent 

Response 

Count 

Internal Organization 

31.3% 

15 

External Organization 

56.3% 

27 

Policy 

18.8% 

9 

No limitation 

12.5% 

6 

Not applicable 

12.5% 

6 

Other (please speeify) 

16.7% 

8 

answered question 

48 

skipped question 

0 


Question 10: Other (please specify) 

1 

It depends. For instanee, if the PM/LSE uses a SETR event that senior leadership/ 
MDA is not familiar with, then it may result in additional exehanges between the 

PM and senior leadership/MDA to educate both sides until both sides agree on how 
it will be addressed and executed. 

2 

Every Navy ‘Heavy-weight’ that eontrols money in the budget thinks they ean 
beeome a eomputer guru by buying the latest and greatest produet that a salesperson 
can talk them into, and don’t seem to realize that all requirements of ANY systems 
should be known PRIOR TO any purchase. Money to spend doesn’t make a 
manager an expert, and they will be rotated out of that job in (usually) three years 
(or less) when someone else will eome in and make deeisions all over again. If you 
don’t know how a present system runs (to fulfill all the laws, polieies, required 
today), then how can you say that you are an expert qualified to replace that system. 

If you foree the eonversion on partial knowledge then the risk of failure inereases 
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and all members of that serviee suffer and Congress is unhappy. Why? Beeause 
they controlled the budget and had the authority but did not have sufficient 
knowledge, or training, or experience, to make the right call knowing the risks. 


Personnel— The organization was not staffed with the appropriate number of 
people. 

3 

Skillset -The work force was not trained for the positions they held. 

4 

PMW 240, as a portfolio of smaller programs, is built around the idea that the 

SETR process needs to be tailored for each, individual program; leadership is also 
aware that defense business systems, especially unmodified COTS 
implementations, are unique among DOD acquisitions, which necessitates 
modifications to the SETR process 


The Program Manager did not and currently does not care for engineers. They “get 
in his way.” 

5 

Maybe that’s why his program is failing...? 

6 

Actual SETR events were determined by MCSC leadership external to the Program 
Office. 

7 

There is very few tailoring guidelines. This leads to significant disagreement about 
what tailoring is acceptable. This problem exists at all levels. Take architecture for 
example: the DoDAE specifications are vague on tailoring, the Navy Architecture 
Development Guide does not directly address tailoring, and the SPAWAR 
architecture review guidelines (used by contractors to conduct reviews) make no 
provision for tailoring. Until tailoring is addressed directly at all levels from policy 
to review guidelines, attempting to tailor will lead to significant friction, frustration, 
and delays. 

8 

The external organizations (user and customer) were not involved in the review. 
Obstacles arose in the approval of a design that the user had never seen and making 
design decisions driving cost and schedule that the customer did not have input to 
change. 


Question 11: Did the program include any metrics to assess the return on 
investment from the reviews? 

Answer Options 

Response 

Percent 

Response 

Count 

No 

80.9% 

38 

Yes (please specify) 

19.1% 

9 

answered question 

47 

skipped question 

1 
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Question 11: Yes (please specify) 

1 

Development, Test, Cost metries 

2 

This should be done on the operating for the thousands of PCs the Navy, of MS 
lieenses versus the eost of Linux type (that ean be make more seeure than MS 
windows). We eould save thousands, if not millions, or billions, and that money 
could go for personnel “perks,” ships and weapon systems if only they looked at the 
long range rather than see how much of a budget they could control, or how much 
money (of that budget) they could obligate and spend! 

3 

Earned Value Management (EVM) 

Action Items tracking, 

Tailored Entrance and Exit Criteria 

Interface prioritization spreadsheet 

Testing Defects per build 

Eunctional and Nonfunctional Requirements per build 

4 

Initial business case and subsequent validation. 

5 

The SETR events were viewed as program milestones necessary for moving 
forward with the next phase of the project. Each SETR event resulted in REAs that 
held the SETR event open until the REAs could be addressed and answered. 

6 

AoAs were run at critical decisions in the program. COAs were used to provide the 
ROI and overall benefits or constraints. Selected CO A implemented with MDA sign 
off. 

7 

We developed a comment review process that tracks comments using our CM tool 
(IBM JAZZ RTC). This allows stakeholders from all sides to see the number of 
comments, where they are in the adjudication process, and when stakeholders feel 
the review is ready to be closed. 

8 

The program had metrics set to meet threshold and objective criteria 

9 

When it came to combining reviews or merging them, then yes. In the past we have 
had several reviews skip the PDR and go to CDR due to multiple reasons. The view 
was that it would save on schedule and allow the team to progress to giving the 
requested solution to requirements vice systematically assessing as we progress 
through the engineering activities. 


Question 12: Did the program have consistent leadership throughout a complete 
SETR cycle? _ 
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Answer Options 

Response 

Percent 

Response 

Count 

Yes 

55.3% 

26 

No 

44.7% 

21 

answered question 

47 

skipped question 

1 


Question 13: Please provide any additional feedback regarding SETR 

implementations. 

1 

Some of the questions set up as “yes” and “no” questions only and do not 
include other or alternative response. There were a couple of questions that 

I was not able to respond with either a “yes” or “no.” For question 11, there 
are metrics used but I was unclear on what was meant by ROI. For question 
12, it was unclear what was meant by “consistent leadership” so I was not 
able to respond. Sorry. 

2 

Not sure if this is applicable to what you are doing. Projects supporting 
various customers, we follow the SIPH MILCON process. 

3 

SETR implementation is a good check and balance to see if processes in 
program is actually implemented as shown in progress. 

4 

SETR in static conditions is very good, but is not suitable for a situation 
where things can change too quickly, where laws and policies require ‘now’ 
(immediate) changes to meet any authoritative mandates dictated. 

5 

The program overestimated the value of the COTS product and tried to 
expedite the requirements phase. The program should have maintained a 
separate Requirements Review vice merging it with Preliminary Design 
Review. 

6 

SETR Events are usually of value added, but in an agile methodology / 
framework environments such events become challenging as the 
atmosphere changes dynamically and with an ad-hoc nature. 

7 

SETR events are not consistent throughout the organization. People 
attending the events have very little value to the process. 

8 

It’s difficult for Tech Warrant Holders to understand entrance & exit 
criteria for Business System SETR event because there isn’t a real tailored 
guideline or acquisition SETR schedule for large business system ACAT 
programs. 

9 

Eunctional community reluctant to adopt modern technology even when it 
was placed at their finger tips. 

10 

In my experience, SETR events often become a “check the box” sort of 
event if the program is required to complete every single SETR review, or 
even to address every single “bullet point” for a given review, regardless of 
how applicable it may be to the given program (if applicable at all). SETR 
reviews are absolutely necessary, but forcing programs to complete reviews 
that are extraneous given their specific situation leads to increased costs 
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(especially on the part of the contractor, especially on smaller programs), 
schedule, etc. 

11 

The AAPs fell under P. Reddy and were done correctly. The ACAT-III can 
use some help. Most team members do not understand the intent or benefits 
of these events. 

12 

Need to have a stick to beat PMs over the head with IRT SETR. No SETR 
should equal a very bad EITREP or something. There just is no teeth to 5.0 
type work and PMs can do what they want...and waste our tax dollars. 

13 

Tailoring SETR events has pros and cons. SETR events typically take a 
block of preparatory time and resolution time to address REAs. This 
typically results in schedule delays. The benefit is that multiple SETR 
events ensure documentation is being updated as the program matures vice 
the documents becoming static and outdated. 

14 

Even though the program was an AAP, the SETR process provided 
disciplined engineering—that provided a product delivered on-time— 
however internal Navy interfaces and dependencies caused delays to the 
overall program. Those programs were NOT using SETR... 

15 

Tailoring provides a costs savings but the most difficult part is convincing a 
PM that is looking at a list that something is not needed or can be a 
simplified version. 

16 

While MCSC has a tech review handbook, different program offices seem 
to handle reviews substantially differently. Some program offices do a two 
week read ahead -i- 1 day review while others do a two week read ahead -i- 
kick-off - 1 - review period -i- comment adjudication period -i- capstone. In the 
latter case, tech reviews seem to stretch out for many weeks/months. We’ve 
captured that time in our process moving forward, but 8-12 weeks seems 
excessive for a tech review. 

17 

We have now moved to different leadership, with an increased focus on the 
administrative aspect of SETRs; participants are expected to review 
applicable documents outside of the actual review event. That does not 
happen—and assuming it will is setting us up for future issues. 

18 

I have none 

19 

It is extremely difficult to determine what is required. Even obtaining 
examples of good SETR documentation seems impossible. Because of this, 
it is difficult to feel prepared and reviews feel more like “gotcha events” 
than the assistance they are supposed to be. 

20 

SETR Is a great method to manage progress of programs 

21 

Program was post Milestone C, but not yet fielded when I joined the team. 
During my tenure, we achieved EOC. 

22 

Managing expectations & frequent sync sessions are critical to success. Ech 

2 & Ech 3 have different interpretations of what is expected regarding the 
level of completeness of each SETR. Often Ech 2 is more concerned about 
a check in the box than ensuring the engineering products are really useful. 

23 

I have the brief for the Agile/SETR process that programs have had success 
in using. However, not all programs are allowed to use that brief. That is 
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due to external organization and not ours. 

24 

I am referring to CANES and the DOT&E oversight of the program. 

25 

SETR process was a good framework for tailored acquisition, but still new 
to the external stakeholders. This created the need to educate people more 
on the modification from the norm. 

26 

Eor a smaller demonstration type program, filling in the gaps created by 
tailoring with the routine program reviews has worked well. A lot of the 
same things are covered, but in a more flexible construct. 

27 

There is often a disconnect between the cost and schedule of a program and 
the goal of the SETR which is more focused on performance. Performance 
is often the bill payer for cost and schedule. 

28 

I have been a part of many types of reviews over the years. Depending on 
the effort, the tailoring was handled differently each time (except for the 
PDR, 9/10 times it is merged into CDR). I treat that action as Eeadership’s 
view to handle SETR reviews and a vision that the workforce is really just 
hardware integrators vice engineers (no real development effort, just 
repackage existing components). 

29 

While SETR is a great tool to assess where a program is in the time of the 
event. We are normally limited in time and resources on how deep to 
conduct the review. We are also limited to the information that the 
programs provide. 

I had an instance where the performers provided all the necessary 
information, everything appeared on schedule for delivery and the SETR 
went by smoothly. I later found out that the performers were 11 months 
behind schedule, but this was not apparent with the provided 
documentation. 

While SETR is a good tool, it is often cumbersome to the programs to 
support this effort while still performing their daily tasks. 

30 

SETR should be a means to implement engage users and customers and vet 
risks before they become issues. 
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